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Rectifiable sets and the Traveling Salesman Problem* 


Peter W. Jones 
Department of Mathematics, Yale University, New Haven, CT 06520, USA 


Oblatum 4-I-1990 


§1. Introduction 


Let K < C be a bounded set. In this paper we shall give a simple necessary and 
sufficient condition for K to lie in a rectifiable curve. We say that a set is 
a rectifiable curve if it is the image of a finite interval under a Lipschitz mapping. 
Recall that for a connected set [ < C, I is a rectifiable curve (not necessarily 
simple) if and only if /(I’) < 00, where /(-) denotes one dimensional Hausdorff 
measure. This classical result follows from the fact that on any finite graph there is 
a tour which covers the entire graph and which crosses each edge (but not 
necessarily each vertex!) at most twice. If K is a finite set then we are essentially 
reduced to the classical Traveling Salesman Problem (TSP): Compute the length of 
the shortest Hamiltonian cycle which hits all points of K. This is the same, up to 
a constant multiple, as asking for the infimum of /(I’) where I is a curve, K < I. 
(Such a I is called a spanning tree in TSP theory.) For infinite sets K, we cannot 
hope in general to have K be a subset of a Jordan curve. What we should therefore 
look at is connected sets which contain K. 

Let Imin be the shortest (minimal) spanning tree. Then we cannot possibly 
solve our problem for sets K of infinite cardinality if we cannot find I, 
I(C) < Col(Fmin), for any finite set K. (Here and throughout the paper C, Cy, C,, 
Co, etc. denote various universal constants.) While there are several algorithms for 
computing /(Inin), these algorithms work for finite graphs satisfying the triangle 
inequality, and do not use the Euclidean properties of K. (See [13] for an excellent 
discussion of some of these algorithms.) Therefore these methods cannot solve our 
problem for general infinite K. We present a method which is a minor modification 
of a well-known algorithm (“Farthest Insertion” — see [13]) which yields a [ with 
I() < Col(Pmin). The Farthest Insertion algorithm has been extensively studied 
with large numerical calculations on computers, and is experimentally good in the 
sense that the I produced satisfy 1(I’) < Col(I'min) for all examples which have 
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been calculated. If K contains N points our method could be modified to calculate 
(up to a constant multiple) /(I’,,,) in time 0(N log N), which is what one would 
expect and which is available for other algorithms [17]. We can thus say that, in 
a certain sense, our theorem gives a geometric “solution” to the TSP. 

A square Q c C is a dyadic square if Q =[ j2~",(j+ 1)2-"] x [k2-",(k + 1)2~"], 
where j, k, ne Z. We denote by /(Q) = 2~" the sidelength of Q. For 4 > 0 we denote 
by AQ the square with the same center as Q, with sidelength //(Q), and with sides 
parallel to the axes. For Q a dyadic square let Sg be an infinite strip (or line in the 
degenerate case) of smallest possible width which contains K 4 3Q, and let w(Q) 
denote the width of Sp. We then define 


o(Q) 
so that B,(Q) measures in a scale invariant way (0 < B,(Q) < 3) the deviation of 
K from a straight line near Q on scale /(Q). Our main result states that K is 
contained in a rectifiable curve I if and only if 


B?(K) = 5) Bk(Q) 1(Q) < © (1.1) 
Q 


where the sum is taken over all dyadic squares (or equivalently over all dyadic 
squares Q with /(Q) < diameter (K)). Furthermore, the shortest possible I has 
length comparable to diameter (K) plus the sum in (1.1). This is an elementary (if 
quite lengthy) computation with the Pythagorean theorem if 6, (Q) < & 9 for all 
Q (é 9 sufficiently small) or if K already lies in a Lipschitz curve (defined in Section 2). 
However, if B,(Q) is large for many Q (K is “dispersed”) there are technical 
difficulties to be overcome. Conversely, if I is a Lipschitz curve it is not hard to 
prove (and is already in [9]) that B?(I°) is bounded by a multiple of /(I). 


Theorem 1. If I < C is connected, then 
B?(T) SCol(L). (1.2) 
Conversely, if B?(K) < 00 there is a connected set ', K < I’, such that 
I(T) S (1 + 6) diameter (K) + C(65) B?(K) . (1.3) 


Corollary 1. If K is an analytic set and 1(K) < 00, then K is totally unrectifiable in 
the sense of Besicovitch ([(4], [5]) if and only if 


B?(E) = 2 
for every E < K with I(E) > 0. 


For certain purposes (see e.g. [1]) it is useful to know that the curve constructed 
for (1.3) can have extra properties. We will show that by taking 6 large enough there 
is [, K < T, such that (1.3) holds, and 


(1.4) Ifz,,z,¢I there is a connected set y c I such that z,,z,¢y and I(y) < Cy 
|z, — Z|. 
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(1.5) If KA Q + @ and /(Q) < diameter (K), there is an infinite strip Sg with 
width B(Q) /(Q), with axis Lo, Sg K OQ + 4, and such that if P denotes 
orthogonal projection onto Lo, 


P(3Q) < P(A). 


Condition (1.4) says that [ is uniformly locally connected and (1.5) asserts that 
I “crosses” Sg 3Q. 

To prove the first theorem we will use a result which is of some independent 
interest. An M Lipschitz domain of size one centered at the origin is a simply 
connected domain whose boundary is a Jordan curve described by r(@)e”, 


1 . ; 
0 < 6 < 2n, where ce < r(@) < 1, and where |r(@,) — r(0,)| < M|e — e|. 


An M Lipschitz domain is a dilate and translate of one of the above domains. 
Lipschitz domains play a central réle in modern one complex variable theory, 
and their behavior vis-a-vis Cauchy integrals and harmonic measure is well 
understood. 


Theorem 2. If Q is a simply connected domain and | (6Q) < o, there is a rectifiable 
curve I’ such that Q\T = |) Q; is a decomposition of Q into disjoint C, Lipschitz 
domains, and j 


¥ 1(@Q;) $ Co 1(2Q). 


We thank Charles Pugh for pointing out that the above theorem fails if we 
replace Cy by &) < 1. Indeed if one tiles (in the above sense) the unit square by ¢, 
Lipschitz domains Q;, ¥ 1(@Q;) = 00. Now let I’ be a bounded connected set and 
attach to I a line segment L and circle S. Applying Theorem 2 to C\(I.U LU S) we 
obtain 


Corollary 2. If I is connected there is a connected set f such that [ cI, 
(Pr) < Cy I(I), every bounded component of C\F isa C, Lipschitz domain, and the 
unbounded component of C\F is the complement of a disk. Furthermore, if x, ye T° 
there is a subarc y < T such that x, yey and I(y) < C,|x — y|. The constant C, may 
be taken to be 3. 


The theorem’s proof ca~ »« modified to yield further structure on I if (1.1) holds 
in some uniform sense. A curve I is said to be Ahlfors-David regular (AD) if 


sup r-'I({zel:s|z—2z| Sr})<o. 


Z EC 
r>0 


Then one can show that K lies in an AD curve if and only if 
sup !(Q)"* }) BRQ')IQ’)<o , (1.6) 
Q Y<e 


and this therefore gives us a characterization of AD curves. If I is an ¢ quasicircle 
passing through 00, Br (Q) < C,e'/? for all Q. Conversely, our construction can be 
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modified to show that if B,(Q) S & < &j for all Q, K is a subset of a d(e) quasicircle 
passing through oo. 

The history behind our proof of Theorem 1 is at first glance a bit surprising, in 
that it was discovered by a careful study of Cauchy integral operators on Lipschitz 
curves. If I is a Lipschitz curve, a famous theorem due to A.P. Calderon [2] (for 
small Lipschitz constants) and Coifman, McIntosh and Meyer [3] later, (for any 
Lipschitz curve) asserts that the Cauchy integral operator is bounded on the 
Lebesgue space L?(I’). B ’ now there are myriad proofs of this result. The theorem 
is very old when I = R, and is equivalent there to the L? boundedness of the 
Hilbert transform. This is essentially easy because one can apply Plancherel’s 
theorem. The author had attempted to find a proof of the result for Lipschitz curves 
by using the intuitive idea that a Lipschitz curve should look like a straight line at 
most places on most scales. This was accomplished in [9] by noting that (1.6) holds 
for Lipschitz curves. ‘ 

Further reflection shows that the above connection between Cauchy integrals 
and the TSP should not be so surprising. Let K have finite one-dimensional 
Hausdorff measure, /(K) < oa An old problem is to decide whether y(K), the 
analytic capacity of K, is zero. In other words, is there a non-constant, bounded 
holomorphic function on C\K? (See [6] or [15] for more on y(K).} An easy 
necessary condition is that /(K) > 0. An old conjecture is that y(K) > 0 if and only 
if Fav(K) > 0, where Fav(-) is Favard length, 


Fav (K) = f I(K,) 40, 
0 


and where K, is the orthogonal projection of K onto the line Re’. This is known to 
be false for sets where K has non sigma finite length [11, 15], but is still open when 
1(K) < oo. The idea Murai has used [11, 15] to attack this problem is to use deep 
estimates on the L? bounds for Cauchy integrals on Lipschitz curves. Now sets 
where 0 < 1(K) < o and Fav(K) = 0 play a central réle in Besicovitch’s theory of 
rectifiable sets (see e.g. [4,5]). Besicovitch proved this occurs if and only if 
I(K AT’) = 0 for every rectifiable curve I’. Such sets are called totally irregular. 
Combining this with Calderon’s theorem on the Cauchy integral [2], it follows that 
I(K) < oo and Fav(K)>0 imply y(K)> 0. This connection between Cauchy 
integrals, geometric measure theory, and the TSP seems all the more natural when 
one considers that the Cauchy integral also plays a vital part in Borsuk’s proof of 
the Jordan curve theorem. In [1] Christopher Bishop and the author use Theorem 
1 and the machinery developed for Cauchy integrals to settle some conjectures 
about harmonic measure. 

The paper is organized as follows. In Section 2 we use an argument with 
conformal mappings to prove Theorem 2. Section 3 uses Theorem 2 to give a proof 
of (1.2). Section 4 is devoted to a construction of the curve [ which satisfies 
(1.3}H{1.5). Section 5 is an appendix where we give an elementary proof that (1.2) 
holds for Lipschitz curves. That result (first proved by Fourier analysis in [9]) is 
used in Section 2. 

The author is grateful to Stephen Semmes for many conversations on condition 
(1.1). I believed at one time that I had a counter-example to (1.3). When Stephen 
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Semmes repeatedly asked me to write it down, I found an error, and Theorem 
1 soon appeared. 


§2. Proof of Theorem 2 


We assume the reader is familiar with corona type constructions as presented e.g. in 
the book of Garnett [7]. Let F: D + Q be any choice of a Riemann map of the unit 
disk onto Q. Since the theorem is trivial if 1(@Q2) = oo, we assume /(0Q) < «0 so 
that F’(z) is in the Hardy space H'. (See [11].) Let G(z) = (F’(z))'/? so that Ge H?, 
and apply either Green’s formula or Plancherel to obtain 


} 1 
J |G(e*) — G(0)|? dd = ff | G'(z)|? logs dxdy . 
T D 
From this we see that if we set F’ = G? = e”, 
1 , 
SSF’ Ilo’ (I? log izl dxdy < 4 {| F’(e*)| dé (2.1) 
D T 
< 81(0Q). 
Now it is well known that ¢ is in the Bloch space B and of norm at most 6, i.e. 
lo’(z)| $1 —|z|?)"*, -zeD. (2.2) 


See e.g. [16]. Let us decompose T into dyadic arcs, i.e. arcs of length 22~". For such 
a dyadic arc I we associate to it a dyadic “square” 


z 
zi 
|2| 


Let Tg = {z€Q: 1 —2n "S|z| S1- n2~"~*} denote the “top half” of Q. Let zg 
be the center of To. 

Fix a dyadic square Q and perform the following stopping time argument. If 
there is z€ Tg with |g(z) — e(zg)| 2 ¢, stop and let Dy = To. In this case we say 
Q is of type 0. Otherwise let Q,,Q.,,.... be those dyadic squares inside Q which 
satisfy 


o-0,-| 


el,l1—22-"<|z|S i. 


sup |g(z) — o(z9)|2¢, 
s0%%, 


and define 
Do 
Then if Dg is not of Type 0, 
lp(z)—@(2g)|S&  zED. (2.3) 


By the construction of Dg we see that Dg = yg is a chord arc domain: 
If z,,2,€7g, there is an arc y C yg joining z, to z, and of length /(y) <6 





P.W. Jones 


Using this procedure we tile D by regions Dg via the usual stopping time 
arguments. If a region Dg so formed is not of Type 0, we say Dog is of Type | if 
(TA yg) 241(T4Q), and we say Dg is of Type 2 otherwise. Then if Dg is of 
Type 1 or 2, 

| F’(z) — F’ (zg)| S 2¢| F’(z9)| 
whenever z€ Do. 
We first bound the Type 0 regions. By normal families and (2.2), 


J IF’ (2)| ds( 5 CIfIF'@) 10")? log = dxdy . 


io Te | 


whenever Do = Tog is of Type 0. Consequently, inequality (2.1) yields 


XY JlF’ Olds) S$ Cl(eQ), 
Type 0 79 
because the regions Zp are disjoint. Now let No be a suitably large integer (in fact, 
No = 2 will do) and divide each Dp = Toe Type 0 into 4° “squares” of essentially 
nae sidelength. Then if Bq i is any of these “squares”, (2.2) shows that F (Do) i is 
a Cy Lipschitz domain. By our last estimate, 


LY LIMOF(GQ)) S$ CoM(6Q). 


We now turn to the regions of Type 1. Since the sets 77 0Q; and TN 6Q, can 
intersect in at most two points if j + k, and since (2.3) holds, 


XY Jl F'@lds(z) 


Type 1 y9 


Y 121+ 2) J | F’(z)\ds(z) 


Type 1 
P TO 


< 12(1 + 2) f | F’(z)| ds(z) 
T 


= 12(1 + 2c) 1(0Q). (2.4) 


The Type 2 regions are a little trickier to bound. The following is the main idea 
of this section: Use the L* bounds of (2.1) to bound the Type 2 regions. Let 
I,,1,,... be the horizontal line segments in yg\0Q. Then by hypothesis, 


me wa 
Pha p2 =p!" Yq). (2.5) 


By normal families and estimate (2.2) there is 6 > 0 such that 
I({ze1;: |g(2) — pg)| 2 9}) 2 S1(L)). (2.6) 


(Because sup |¢(z) — 9(Zg)| 2 &.) Now let w denote harmonic measure for the 
zeT(Q;) 
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region Dg with respect to the base point Zo, and let g(z) denote Green’s function for 
Dg with pole at zg so that 


Then since Y, is a chord-arc domain, the results of Jerison and Kenig [8] show 
that w is an A,, weight, i.e. there are constants A, 7, yn’ > 0 so that 


-1( (E)\’ EY" 
? Gey) aGrs Are) ss 


for any Borel set Ec yg. (The usual definition of A, uses only the second 
inequality, but it is well known that this is equivalent to the first [7].) Combining 
(2.3) with (2.5}{2.7) we see that 


J | G(z) — G(zg)|? dw = c| F’(zg)|, 
where c = c(6, A, n) > é Green’s formula is valid for chord arc domains, so that 
2n J | G(z) — G(zg)|? dw = ff |G'(z)|? g(z) dxdy . 
Ye Gq 
An elementary argument with the maximum principle shows that if J = Tq éQ, 


gle) < CWI)? log 


1 : ; Pie : 
whenever z€ Yg and |z — Zg| 2 16 I(1). Applying the maximum principle again we 
obtain 


J | F'(z)|ds(z) S Cl (yg) J | G(z) — Glzg) |? dw 


@ Ye 


1 
S Clq) {J1G@)? Wy9)"" log dxdy 
% 


1 
=C’ JflIF Ile!’ log 5 axdy 


z 7 


Since the regions Zp are disjoint, it follows from (2.1) that 


| 
XY J | F@lds2) < C fl F119’)? log —dx dy (2.8) 


Type 2 7 |z| 
<C’'l(éQ). 


This almost finishes the proof of the theorem, because if ¢ is small enough each 
domain F (YQ) will have boundary a (7) chord-arc curve. (This follows immediately 
from (2.3).) Furthermore by estimates (2.4) and (2.8) on the Type 1 and Type 
2 domains, 

¥ M(GF (Bg) S C# (AQ). 
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We thus need only show that each Type 1 or 2 region. Yg can be decomposed into 
disjoint Cy Lipschitz domains Q; with }’ 1(02,) < Col (yg). For then F (Q,) will be 


J 
a C, Lipschitz domain if ¢ is small enough, and 


¥GF(Q,)) < CIF (De). 


The construction is a little easier to describe on the upper half plane 
R2 = {z=x + iy: y > 0}. We are given a dyadic square Q < R3 with side Ic R. 
The region Zg is constructed by removing from Q disjoint squares Q,,Q,, . . . with 
(disjoint) sides I; < R. Let I; denote the top side of Q;. We now build a Cantor 
type tree T;. Put J; in T; and add on line segments on the left and right of J; down 
to R and of length a 1(1;). We obtain a nice “tree” T} with two “roots” of length 
/2 I(I;) touching R at angle 7/4. Let z, and z, be the two points in T. + of height 
1/4 1(1,) so that Re z,; = x; = k,(1/41(1,)),k,;€Z,j = 1, 2. (In other words, x, and x, 
are appropriate dyadic rationals.) Attach to z, and z, the line segments to R (of 
angle 7/4 with R) which are not already in 7} and call this 77. 77 is obtained by 
sprouting the roots of 7}. Continue sprouting, by dropping by a factor of 1/4 each 
time and obtain 7}, 77,..., 77, . .. Then 77 has 2” “roots” of length is 4°" 1(1;) 


and each terminates at a point k 4~" /(1,) where ke Z. Setting T; = |) Tj we have 
k=1 


K(T;) = (1 + 3,/2) (1). 


It is an exercise to see that every component of Dg\\) T; is a Cy Lipschitz 
domain. 0 j 


We remark that the idea of using Littlewood—Paley estimates to impose bi- 
Lipschitz structures on sets has been used before in [10] to give a certain quantitat- 
ive version of Sard’s theorem. 


§3. Proof of (1.2) 


We may assume that /(I) < o and the conclusions of Corollary 2 hold for I, 
because if < F, B (I) S B? (I). We denote by Q; the components of C\I and set 
,=0Q;, d;=diameter (Ij). For a dyadic square Q let F(Q)= {Tj: 
Tr; 4Q + $,d,; 2 1(Q)} and let G(Q) = {I';:',; 15Q + ¢,d; < 1(Q)}. Also let Q* 
be the dyadic double of Q. 


Lemma 3.1. 
BF(Q) SC, Y BF(Q*)+C,1(Q)-? Y Area(Q,). 


F (Q) G(Q) 
Proof. The lemma is immediate if #(Q)= 4, for then either B(Q) =0, or 


y. Area (Q,) = 91(Q)?. So suppose I, € F (Q). By scaling we may assume /(Q) = 1. 
GQ) 





Rectifiable sets and the Traveling Salesman Problem 


Let L be a line such that 


d= sup distance (z, L) < B,, (Q*)1(Q*). 
zelyn5SQ 
Let z)9 €T 1 3Q maximize distance (z, I) and let distance (zp), [y) = dy. Denote by 
z, the closest point in I’) to zo, and let I = [zo, z, ]. We define z, = 1/2(z) + z,) to 
be the midpoint of I. 


Case 1. F (Q) contains three or more curves. Then there is [;¢F (Q) such that 
Br (Q*) = C,, so 
2 B7,(Q*) P-4 C, . 


F (Q) 


. d Bs 
Case 2. F (Q) contains only I’). Then a( =.) < |) Q,. Consequently, 
G(Q) 


Br(Q) S$ (d + do 
S 2fr,(Q*) + 2d6 


<2H2,(0%) +7 ¥, Areal). 
G (Q) 


Case 3. F (Q) = {I, AT)). 


We may as well assume f,,(Q*) < &, j = 0, 1, for otherwise there is nothing to 
prove. Letd, = sup distance (z, I,). Then if ¢, is small enough I 7 3@ is trapped 


zel,n4Q 


between I, and I, and 
Br(Q) S Br.(Q*) + Br, (Q*)) + 4, . 


Since I, is a Cy Lipschitz curve there is z,€9/2Q\(Q, UQ,) such that distance 
(z3, Ij) 2 cod,, j = 0, 1. But then 


Br (Q) S$ Co(B7,(Q*) + BF, (Q*)) + Codt 


S Co(BF,(Q*) + BF,(Q*)) + C, } Area(Q,). 
G(Q) 


It is now an easy matter to finish the proof of (1.2). By the results of [9] (or our 
section 5), 


Y BF, (Q)1(Q) < C, 1) 
Q 


for any Cy Lipschitz curve. By Lemma 3.1 it is enough to estimate the sum 
¥1(Q)"* ¥ Area(Q,). 
Q G(Q) 


Now for each neZ such that d; < 2~" there are at most C, squares Q such that 
(Q) =2~" and I';€G(Q). Consequently, we may estimate the above sum by 
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reversing the order of summation to obtain 


¥ Area (Q,) Y 1(Q)7' 
Ty Q 
2,€G(Q) 


(2) 


(2"d,)* 
=0 


< CY Area (Q,) 
Tx 


< 2C¥ Area (Q,) dy! 


Tr 


SCYMUP,) S$ 2C UL). 


§4. Construction of I’. (Proof of (1.3)-(1.5).) 


We build sets Yo, £,,...,L, < K with the following properties: 


lz;—- 41 227%, 2,%,€L,, j#k. 

and 

inf |z—z,;| $27", zeK. 

z,€f, 
These sets may need to be slightly perturbed at various stages of the construction, 
but the two properties listed above will still hold. We may assume K c [0, 1]?, and 
by scaling we may also assume Y, = {2Z9, z,} where |Z) — z,| = sup |z — w|. We 

z,weK 

define I) = [2z) — z,, 22, — Zo] to be a line segment containing Zz, z,, and ex- 
tending beyond those points. We let A = 1 be a constant to be fixed later, A = 2", 
and then let &) >0 be small. The value of ¢) is determined later. Suppose by 
induction that Ij, [',,...,I,-, have been formed and let x,e¥,\/,_,. If 
Xo €I,,_, we do no construction about xy. Let Q be that dyadic cube containing x, 
with 1(Q) = A2~". We call the collection of all such cubes J,. By a translation and 
rotation we may assume xX, 2 0 and its nearest neighbor in Y,,_ , is the origin. Let 
W = {z: 0<|z| S$ A2~"*', |argz| < 2/3}, and let W* = {z: 0 <|z| < A2™"*}, 
\n-argz| < 27/3}, so that WU W* = D = {z: 0 < |z| S A2~"*"}. We assume by 
induction that the following properties hold: 


(Pl) Let {y,,...,y¥y} =Y%,-, {Dv {0}} and arrange the points so that 
Rey, S Rey, S$... Reyy. Then I,_, contains the segments [y,, y;.,], 
1<N-1. 


(P2) If xo¢F,-, and ¥%,_,7AW=® there is 0, |@|< 27/3, such that 
(0, A2-"*'e®] cIr,_,. If xo¢I,_-, and Y,_, 1\W* = 4, there is y, 
|W — 2| < 2n/3, such that [0, A2-"*'e*] cT,_,. 


Case 1. B(Q)2& . Connect x, to all points in #,A{|x —x |< C2™"} by 
(straight) line segments. Also add on the line segment [0, 44x, ]. Then the amount 
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of length added to I’,,_, is bounded by 
cy ¢@sc ¥ KOI) 


Qe9, Qe9, 
because each Case 1 Q has B(Q)? = ¢. 
For the rest of the cases we assume £(Q) < é. 


Case 2. £,_,\W+, £,_, 0 W* = &. Let y, € W minimize |z|, ze Z,_, 0 W, 
and let y_ , € W* minimize |z|,z¢Y,_, 0W*. Let {x,, .. . ,X,¢} be those points in 
L,AW such that Rey_, <Rex,;<Rey,, and label the points so that 
Rex, S Rex, S ...< Rexy. Replace the segments [y_,, 0], [0, y, ] < [,-; with 
[x,, X2],..., [xm-1, Xy]. Then by the Pythagorean theorem, the amount of 
length added to I’, is bounded by 


CB? (Q)1(Q). 
Since B(Q) < &9, properties (P1) and (P2) are maintained. 


Case 3. £,_,ANW+o2 £,_,0W*=6,2,0W*a{\z| <5 27-"*!} =. Let 
y,EL,-,OW minimize |z|,ze%,_,A1W. Then ([0,y,]cTI,-_,. Let 
{0,X1,X2,..-,Xy,y;, } be the points in Y, 7 W between 0 and y,, arranged by 
increasing real parts. Replace [0, y, ] by [0, x, ], [x,, x2], ...,[%y, y, ]. Then by 
Pythagoras the amount of length added is bounded by 


CB? (Q)1(Q), 


and properties (P1) and (P2) are maintained at x,, . . . ,Xy because B(Q) < &). We 
now add on some line segments near 0 and y, to assure properties (P1) and (P2) 
hold there. First suppose x_ , € K 7 W* maximizes |z|, z€ K 0 W* cq {|z| < A2~"}. 
If |x_,| <8A~12~"add to I, the line segment [ — 2~"*', 0]. If|x_,| 28A~*27", 


a o| In either case the amount of length 


n+1 


add to I’, the line segment [2 


|x_, 
added is bounded by 2~"*'. Let zo, z,,...,Zy =X, be the points in 7, ,,,.9 {| 
z| S$ 2~"}, arranged by increasing real parts, where 2*° = 2A*. We add on to I’, the 
line segments ([Zo,2,],...,[Zy-1,Zy], SO that at stages n+l, 
n+2,...,n+k,—1 no constructions need be performed in {|z| < 2~"}. Since 
B(Q) < 9, properties (P1), (P2) are preserved for z,,...,Zy at stage n + ky by the 
X-1 


choice of [ — 2~"*',0] | respectively 2~"*? bat 
=—s 


,0 |. A similar construction is 


performed at y,. 
Let E = [0, A2~"*' e”] <T,_, 0 W* be the line segment assured by hypo- 
thesis (P2) and let 


Ig = [A2-"~* e”, A2-"e”]. 
Then the construction will show that Ig is not altered at any future stage, 


Ig < () Vysy- If A is large enough, 
k=0 


(4.1) The amount of length added is bounded by 1/2 /(J9). 
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Furthermore, if Ig and Ig, are any intervals so formed at any stage of the 
construction, 
IgnIg =¢. (4.2) 


Case 4. £,_,AW+9,f,_,0AW*=6,£,0W*a{|zZ| 5 2°-"*'} +. First 
suppose %, 0 W* a {|z| < 2-"*!} = {x_,}. We may assume, by changing Y, if 
necessary, that x_, maximizes |z|,z€K 0 W* cq {|z| < 2~"*"}. The construction 
is as in Case 3, but we add the segment [2x_,, 0]. Then estimates (4.1) and (4.2) 
hold and as in the argument of Case 3 properties (P1) and (P2) will hold at future 
stages. If Y,, a W* m {|x| < 2~"*'} contains two points x_,, x_., we may assume 
xX_, maximizes |z|,z¢eK 01 W* q {|z| < 27"*'}. Then we add [2x_,,x_,], and 
[x_,,X_2], [x_2, 0]. Estimates (4.1) and (4.2) hold as do properties (P1) and (P2) 
at future stages. The case where Y, 0 W* 4 {|z| < 2~"*"} contains two points is 
treated similarly. 


Case 5. £, ,;,AW=060 £,_,\1W* +d. We assume x. maximizes |z|, ze 
KAWoa\{\|z| S$ 27"*"}. Let {y1, y2,-- Yn = Xo} be the points in (LY, W*) U 
{|z| < 2~"**} arranged by increasing real parts, and as in Case 3 replace arcs of 
I,,-, in that region by [y,, y2],...,L¥n-15 YyJ- As in Case 3 we add on [Xo, 2x ] 
and line segments in Wn {|z| S$ 2~"*"} so that Zx4,, .WO {|zZ| S2°-"*1}3.CT,,. 
We also choose Ig as in Case 3. Then (4.1) and (4.2) hold and as in Case 3, (P1) and 
(P2) are preserved. 


Case 6. £,_;AW=9,f,_,1W* =. Let {y,,...,yy} be the points in 
Leto O{|z| <2°>"*"} arranged by increasing real part. We may assume either 
y, =0 or y, maximizes |z|,z€K 0 W* 9 {27*) < |z| <2-"*"}, and we may 
assume yy maximizes |z|,z¢K Wo {|z| <2~"*'}. Add on the line segments 
[2y1, yi], [2yn, yw] and [y;, yj+1], 1 Sj SN — 1. Let Ig be as in Case 3. Then 
(4.1) and (4.2) hold and (P1) and (P2) are preserved. 


Remark. By the choice of I’, only Case 1 or Case 2 constructions can happen at z, 
and z, until stage kj, 2° = A. Therefore (P1) and (P2) will always hold at zp, z,. 


To conclude the proof we note that by Cases 1-6 and estimate (4.1) the quantity 
I(T) —1(T,,-,) is bounded by 


CY BQO) +5 Y Ml). 


QeF, QeEZ%, 


Summing from n = 1 to N we obtain 


1 
(Ty) — 1(Io) S CL BO) (Q) +5 y Ig) 


(Q)s2-" 


1 
S CL BQO) + 5x) 


the final inequality being a consequence of (4.2). Therefore 


I([y) S$ 21(P) + C YB? (Q)1(Q), 
Q 
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and taking limits we obtain the first part of the theorem. We note that the segments 
[229 — 21,29], [z,, 22, — Zo] < Ty are never altered at any stage of the construc- 
tion. By throwing them away and taking A large enough we could build I so that 


I(I') S$ (1 + 6) diameter(K) + C; >) Bk (Q)/(Q), 
Q 


but then (1.5) would not hold for Q = [0, 1]?. 

To show property (1.5) holds, we must add some line segments to I to form 
a new curve I’. Fix a dyadic cube Q with B(2Q) < €, and first suppose that there are 
points x»€¢K 0 Q and x,,x,¢€K 7(5Q\3Q) with the angle between [x,, x, ] and 
[x9, x2] greater than 2/2. Then the construction yields a subcurve of I which 
connects x, to x, in Sg. For the other case (where x, , x, € K 7 (5Q\3Q) implies the 
angle between [x,, x, ] and [xo, x, ] is less than 2/2), the construction shows there 
is an arc Ig < I: 3Q such that distance (Ig, K) 2 C 1(Q). Add to I a line segment 
Jq crossing 5Q in So. Then I(Jg) S Cl(Ig) and since the Ig are essentially disjoint 
from each other, 


E19) <CYMIg) SCP). 
Q 


Consequently | (I°) < C I(I). 
~1o show that (1.4) holds, apply Corollary 2 to [ and obtain i such that 
rr )SCot+ C, B?(K) and such that the conclusions of Corollary 2 hold. Let 


C\F = Q,, and suppose x, yer. Let I =[x,y] and let I= E\)\J I; be a 
= i 
decomposition of J into E = I VT and open intervals I; which lie in Q, ; ;. Setting 


I= Lx; y;], we have x;, y;€0Q,,j), and consequently there is an arc WS 02; ;) 
connecting x, to y; such that (y;) < Cy|x,; — y;|. Then if y = E\ |) y;, 7 is connec- 


~ J 
ted, y < I’, and I(y) S$ (E) + ¥ I(y;) S$ WE) + Co Y |x; — yj S$ Colx — yl. 
j j 


§5. Appendix 


In this section we show that (1.2) holds when I is a Cy Lipschitz curve. By using 
a dilation, we may assume that I is given by the parametrization w (6) = r(@)e”, 
where Co! < r(6) < 1, and |r(0,) — r(8,)| < Co|6, — |. Let I, be the polygon 
obtained from the line segments 

Ji =(Ww(j2-"** 2) WG +12"), «= Sj S2". 
Then J} splits into two intervals J37*, J3j}, at stage n + 1. Define 


On,j=2°" sup distance (z, J). 


ae Nina 
Then by elementary geometry, 


U(JS25*) + U(I374n) — USP) 2 Cn, 2 
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Summing from n = 1 to oo we obtain 


oY (y,j22" SUP). 


y Qasr t scr”. 


k=n Je cop 


Here we define PF? = [a {j2-"*'n<O<(j+ 1)2-"*' 2} = OG}. Our result 
will follow if we can show that 


B(r?) = 2-" sup distance (z, J?) 
zel? 


satisfies 
Leys “Set, . (5.2) 
n,j 


for then we may rotate the dyadic grid through [0, 27] to obtain quantities B, (I 7) 
and note that 


Y B(Q)? 1(Q) < 


1(Q) ma 2-0-2 
2n 


Cf {EBury2-*} ao. 


To prove (5.2) notice that 


pS, ye. 2°". 


k=1 Ink cor 


Then (5.2) follows from the above inequality, Minkowski’s inequality (or Cauchy— 
Schwarz), and (5.1). 
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1. Introduction 


1.1. Let p be a prime number. In [2] Broué and Puig introduce the so-called 
nilpotent blocks: a block b of a finite group H is said to be nilpotent if, for any p- 
subgroup Q of H and any block f of C,(Q) associated with b, the quotient 
Ny(Q,f)/Cy(Q) of the stabilizer of (Q, f) in H by the centralizer of Q is a p-group. 
In [18] Puig completes the study of such blocks giving the structure of their source 
algebras. As the title announces, the main purpose of this paper is to study the 
structure — i.e. the source algebra, in the sense of [17] and [18] — of any block of a 
finite group G lying over a nilpotent block b of a normal subgroup H of G. 


1.2. Such a purpose has been motivated by the following situation, which occurs 
in any block a of any finite group G: consider a p-subgroup Q of G and a block f of 
C,(Q) associated with a; if Q is “big enough” — precisely, if Q is self-centralizing in P 
for any Brauer pair ( P, e) containing (Q, f) (cf. [1]) — then Z(Q) is a defect group of 
f, in that case, f is clearly a nilpotent block of Cg(Q) and therefore our results 


* This work has been partially supported by the DFG and the CNRS 
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determine the structure of the block of N,(Q) lying over f. The particular case 
where Q is a defect group of the full block a of G has been already considered by 
Reynolds in [21], by Kiilshammer in [10] and by Puig in Proposition 14.6 of [19], 
where the source algebra of the block of N,(Q) lying over f is completely 
determined. 


1.3. On the other hand, if Ng(Q) is p-solvable, the situation above could be 
handled employing Fong’s reduction techniques in [8] — suitably extended to 
discuss algebra structures as described for instance in [13]. However, notice that 
our results here are more precise even in this case since they guarantee the 
“uniqueness” of the final reduction, which is important to connect the structures 
coming from the different pairs (Q, f). 


1.4. Let us come back to our basic situation. Let 4 be an algebraically closed field 
of characteristic p, 0 a complete discrete valuation ring with residue field 4, G a 
finite group, H a normal subgroup of G, b a primitive idempotent of ZOH, N the 
stabilizer of b in G, P a maximal p-subgroup of N such that Brp(b) + 0 in Z4C,,(P) 
and Q the intersection of P and H. Our main results state that, if the block b of H is 
nilpotent, the structure of any block of G lying over b is completely determined by 
the following two invariants 


1.4.1. a full O-matrix interior P-algebra S which has a P-stable O-basis containing 
the unity (i.e. a capped endo-permutation ©P-module, in Dade’s terms cf. [6], S 
being its O-endomorphism ring). 


1.4.2. a twisted group algebra 0,L constructed from a group extension L of N/H by 


Q, which contains P as a Sylow p-subgroup. 


Actually, Theorem 1.8 below shows that the uniqueness of L is stronger than the 
uniqueness expected from a group extension of N/H by Q: the effort to obtain that 
uniqueness — which is very important for forthcoming applications — lengthens 
particularly this paper. Let us mention an easy case: when Q has an N-stable 
complement K in H - which is the situation considered by Dade in [7] — L is just 
the quotient N/K. 


1.5. To state precisely these results, we will freely use notation and terminology 
introduced in Sect. 2. Set A = OG, B = OH, B = {b} so that B is a point of H on B 
and N = N,(H,) (cf. 2.7). For any X < N denote by X the image of X in N/H; in 
particular, N = N/H and we consider N, endowed with the action on the set N 
given by left multiplication, as an N-permutation group (cf. 2.4). We consider A 
endowed with B as a G-algebra extension — an interior G-algebra endowed with a 
G-stable subalgebra containing the unity (cf. 2.9) — but, as we explain below, we are 
mostly interested in the restriction Res§(A) (cf. 2.9). Notice that is still a point of 
N (and of any subgroup of N containing H) on B and that H, < N, (cf. 2.7). 


1.6. Let « be the unique point of G on B such that Hy CN, cG, (ie. «= 
{Tr§(b)}), y a local point of P on B such that P, < Ng ¢ G, and 6a local point of Q 
on B such that Q; < P,. Notice that y and 6 exist since Br3(b) + 0 and therefore 
Br6(b) + 0 (cf. 2.6 and [1], §1). Moreover, P, and Q, are respectively defect pointed 
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groups of G, and H, since our maximal choice of P guarantees that P, is a maximal 
local pointed group on B contained in N,, so that ac Tr§$(B”-y-B?) 
(cf. 2.7), and then Q; is a maximal local pointed group on B contained in H, (see 
Proposition 5.3 below for more detail). Consequently, we get N = H- N,(Q;) by 
the Frattini argument (cf. [19], 2.10.2) (we have Ng(Q;) < N since B is the unique 
point of H on B such that Q,; < H,); in particular, if the block b of H is nilpotent, 
the inclusion N¢(Q;) < N induces 


N = N¢(Q5)/Q°Cy(Q) = Ey, w(Qs) (1.6.1) 


since Ny(Q;) = Q°Cy(Q) (cf. [18], 1.7.2) and Cy(Qs, N) = Cy(Q) (cf. 2.7). A 
general remark: if H = Q then all the points above coincide with {1} and we will 
omit to mention them. 


1.7. We denote respectively by f,: A,>A, fg: Ag—Res¢(A) and f-: A,>Res§(A) 
embedded algebra extensions associated with G,, N, and any local pointed group R, 
on B such that Q,- R,c P, (cf. 2.11). Notice that, as bxb=0 for any 
xeEG—N, the embedding 7A induces an N-algebra extension isomorphism 
A, = ONb where ONb is endowed with OHb (cf. 2.9), and therefore a G-algebra 
extension isomorphism (cf. 2.12.2) 


A, = Ind$(ONb) (1.7.1) 


which shows that, for our purposes, it suffices to consider A,, and a fortiori 
Res{(A). Actually, as we explain from 1.16 on, it suffices to study the P-algebra 
extension A, and Theorem 1.12 below describes its structure, which depends on a 
suitable group extension L of N/H by Q as we said in 1.4. We first give our main 
result about L. 


Theorem 1.8. With the notation above, assume that the block b of H is nilpotent. 
There are a finite group L and group homomorphisms 1: L — N and t: P > L fulfilling 
the following conditions: 


1.8.1. We have Im(x)=N, Ker(z) = t(Q), Ker(t) = {1} and n(t(u)) =a for 
any ueéP. In particular, considering L endowed with the action on the set N given 
by x and left multiplication as an N-permutation group, t induces a bijection 
Tr: Hom;(R, P) ~ Hom ,(t(R), t(P)) for any subgroup R of P. 


1.8.2. For any local pointed group R, on B such that Q;- R, < P, we have 
Tr(En, w(R,, P,)) = E,, a(t(R), t(P)). 


Moreover, if L’ is a finite group and m':L'+N and t':P-—L’ are group 
homomorphisms fulfilling conditions 1.8.1 and 1.8.2, there is a unique group exo- 
morphism 6: L = L’ having a representative o such that n'oo = 1 and oot =T7. 


Remark 1.9. Notice that t(P) is a Sylow p-subgroup of L (cf. 1.8.1, 1.8.2 
and 2.16.2). In particular, any group automorphism @ of L such that row=2 
and wet =T is an inner automorphism determined by an element of Co( P) (cf. 
Remark 4.12 below), which proves the uniqueness of & above. 


Remark 1.10. In Proposition 5.5 below we show that ¢ is the unique local point of 
R on B such that R, ¢ P,, and therefore if 7, is another local pointed group on B 
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such that Q; < T, < P, we have 
tr r(Ey, w(Ty, R.)) = Ex, x(t(7), (R)) (1.10.1) 
where Tr. z is the bijection from Hom n(7, R) onto Hom n(t(T), t(R)) induced by t. 


1.11. From now on, we assume that the block b of H is nilpotent. To construct A, 
from the group L above, we have to introduce a suitable central extension of L by 
£*, obtained by restriction from the following central extension of N by 4*. As b is 
nilpotent, B,/J(B,) is a simple k-algebra (cf. [18], 1.9.1); denote by s: B, > B,/J(Bg) 
the canonical map and, in particular, set B,/J(B,) = s(B,). The natural action of N 
on s(Bz) defines a group homomorphism N — Aut(s(B,)) and, considering s(B,) * 
as a central extension of Aut(s(B,)) by 4”, we set (cf. 2.1 and 2.4 below) 


N = Resy(s(B,)*) ; (1.11.1) 


that is, the elements of N are the pairs (s(c),x) where xeN and ceB; fulfils 
s(d°) = s(d*) for any de By. Moreover, there is a unique group homomorphism 
HN mapping x€H upon (s(x- 1), x), this homomorphism is injective and its 
image is a normal subgroup of N intersecting trivially the image of #*. So, 
identifying H with its image in N, the quotient N/H is a central extension of N by 
£* denoted by N. Recall that, if L is a central extension of a finite group L by 4“, we 
denote by 0,L the corresponding twisted group algebra (cf. 2.1 and 2.4 below) and 
by 0,L° the opposite @-algebra. 


Theorem 1.12. With the notation above, assume that the block b of H is nilpotent. Let 
L be a finite group and x: L > N and t; PL group homomorghiane fulfilling 
conditions 1.8.1 and 1.8.2, set L = Res, (N ) and consider 0,L°, endowed with the 
group homomorphism P > (0, L°)* induced by t and with the image of OQ, as a 
P-algebra extension. Then there is an O-simple interior P-algebra S, unique up ‘to 
isomorphism, such that we have an isomorphism 


A, = S@0,L° (1.12.1) 
© 


as P-algebra extensions, and the determinant maps the image of P in Ind&p)(S")”, 
where S* is the restriction of S to t(P), onto {1}. Moreover, S has a P-stable ©-basis 
containing the unity as unique P-fixed element. 


Remark 1.13. If S’ is an O-simple interior P-algebra such that A, = S’@,¢ 0,L°, 
there is a group homomorphism 6: L > ©” such that S’ = Res,(,) @@S where 
©, denotes the corresponding interior L-algebra. Indeed, it is clear that 
0, @o0, DY = O, [° as P-algebra extensions and, since p divides neither rank,(S) 
nor |L: «(P)| (cf. Remark 1.9), we may choose @ such that the determinant maps the 
image of P in (O, — 1 @@Indtp,(S'*))* onto {1}. 


Remark 1.14. Since 1.12.1 maps B, onto S @~0Q and P*1 onto {(u-1)@u},cp, 
this isomorphism induces a P-algebra extension isomorphism from }',_p»u°B, 
onto S@,OP. Hence, since f, induces an embedding from }),.pu°B, to 
yep 4° B, = O(P*H), S @g OP becomes a source algebra of b as a block of P-H. 
This shows that b is still a nilpotent biock of P « H (cf. [18], Th. 1.6) and therefore 
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proves already the uniqueness of the P-algebra S and the existence of a suitable P- 
stable (-basis of S (cf. [18], Th. 1.6). In particular, as Aut(A,) stabilizes )) pu’ B,, 
we have a group isomorphism 


Aut(0,L°) = Aut(S @ 0,L°) (1.14.1) 
O 


mapping ge Aut(O,L°) upon id, ® g (cf. [18], Cor. 6.9 and [15], Prop. 2.1). 


1.15. When Q is not normal in H, it is useful to know the following alternative 
definition of N', which depends on a theorem of Dade (cf. 2.12.4). The action of 
N,(Q;) on the simple k-algebra B(Q;) (cf. 2.7) defines a group homomorphism 
N,(Q;) > Aut(B(Q;)), and then, considering B(Q;)“ as a central extension of 
Aut(B(Q;)) by 4“, we claim that (cf. 2.4). 


1.15.1. there is a group extension isomorphism 


Resy.(9,(N) = Resy..9,(B(Qs)*) 
mapping (s(vz- 1), vz) upon (ss(z), vz) for any ve Q and any z€Cy(Q). 


_ This isomorphism provides another definition of N; indeed, we have 
N = Ng(Q5)/Q *Cy(Q) (cf. 1.7.2) and, identifying Q-C,(Q) with its image in 
Resy (9,)(B(Q5)*) through the injective group homomorphism mapping vz upon 
(s5(z), vz), 1.15.1 induces a group extension isomorphism 


— 


N & Resy.9,(B(Qs)*)/Q*Cu(Q) . (1.15.2) 


Set Resy_(9,(N) = N(Qs); to prove 1.15.1 it suffices to exhibit a group 
extension homomorphism N,(Q;) > B(Q;)* extending the group homomorphism 
Cy(Q)— B(Q;)* induced by s;. We have already a group extension homomor- 
phism 


Ng(Qs) > s(By)* (1.15.3) 


extending the group homomorphism C,(Q) —> s(B,) induced by s (cf.1.11). More- 
over, the canonical homomorphism 4 ®, B, > s(B,) determines a strict covering 
exomorphism of interior H-algebras over 4 (cf. [18], 4.14); indeed, setting also 
s(B;) = B;/J(B;), it follows from Theorem 1.6 and Example 4.25 in [18] that the 
homomorphism 4 ®, B;— s(B;) determines a strict covering exomorphism of 
interior Q-algebras over k (since b is a nilpotent block of H), and therefore it follows 
from Theorem 3.8 and Corollary 4.23 in [18] that the induced homomorphism 
Ind§(4 @~ B;) > Ind§(s(B;)) determines a strict covering exomorphism of interior 
H-algebras over 4 (since H is locally controlled by Q on 4 ®@ B;, so on s(B;)); now 
the statement follows from Theorem 3.4 in [15] and Proposition 4.17 in [18]?. 
Consequently, s(d) is contained in a local point 5 of Q on s(B,) (cf. [18], 4.14 and 
4.19), and we have an N,(Q;)-algebra isomorphism (cf. [18], 4.14.2) 


B(Qs) = s(Bg)(Q5) (1.15.4) 


1 This alternative definition is only employed here in 1.20 below to show a consequence of 
Theorem 1.12 

2 This argument reproduces line by line the first part of the proof of Proposition 7.2 in [18], and 
the reader may refer himself there for more detail. 
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and a group extension isomorphism (cf. 2.12.4, [18], Th. 1.6 and [19], 6.22) 
F p,)(Q3) = Fyp,(Q3) = 4** Fyp,(Q5) - (1.15.5) 


Now it suffices to compose the group extension homomorphism 1.15.3, whose 
image is contained in NyB,)* .(Oz) where QO is the image of Q in s(B,)*, with the 
group extension homomorphism Ns(B,)* «(O;) + B(Q;)* obtained from 1.15.5, 
2.12.3 and 1.15.4. 


1.16. We will explain now why Theorem 1.12 provides complete information on 
the blocks of G lying over b, as announced in 1.7. First of all, by 2.14.1 there is a 
unique interior G-algebra embedding 


gi: A, > Ind$(A,) (1.16.1) 


such that Res$(g?) of a. 4G p(A,), and therefore the local pointed groups on A, are 

4 the G-conjugates of local pointed groups on A, (cf. [17], 1.12). Moreover, as 

= S@_O, L° as interior P- algebras (cf.1.12.1), for any subgroup R of P there is 

a " bijection ‘LA, (R) = ZPo jo(R) mapping «°c LP,(R) on the unique 

t(e°)€ LP o, fo(R) such that en + 0 (cf. 2.3 applied to 1 @ (0, [Rc (A,)¥, 
and [18], Th. 5. 3), and considering 0, L° as an L- algebra we set 


t(R)qq0) = t(R,o) - (1.16.2) 


So the local categories of A, and 0,L° (cf. 2.8), and the local fusion category of A, 
(cf. 2.12) have, up to isomorphism, the same objects and the following statement 
shows that they have also the same morphisms. 


1.16.3. If Ro and Tyo are local pointed groups on A, then Eg(Tyo, R,o) = 
F 4 (Ty, R ) and tr, (Fy (Tyo, R o°)) = Ex(t(Tyo), t(R,o)) where typ is the 
bijection from Hom(T, R) onto Hom(t(7), t(R)) induced by t. 


Indeed, by Theorem 3.1 and Proposition 2.14 in [17] we know already that 
Eg(Tyo, R,o) = Fa(Tyo, R.o) = F4 (Tyo, R,) 
and identifying t(P) with its unique lifting to L°, that 
E,(t(Tyo), (R,o)) = Fo, z0(t(Tyo), t(R,»)) 


(since, by Lemma 5.5 and Proposition 5.15 in [19], 0,L° can be embedded in a 
group algebra OL’ of a finite central p’-extension L’ of L). Moreover, by 1.12.1, 
O,L° can be embedded in S° @, A, (since © can be embedded in S° @,S) and 
therefore, again by Proposition 2.14 in [17], we have 


Fo, jo(t(Tyo), t(R,0)) = Fs @ 4, (t(Tyo), t(R,0)) ° 
© 
Now, 1.16.3 follows from the following general result. 


Lemma 1.17°. With the notation above, let D be an interior P-algebra having an O- 
basis X such that P- X- P = X and|P*x| =|P| =|x- P| for any xe X. If R, and T, 


3 This lemma is not employed in the sequel. Actually, it should have been already in [18], §5! 
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are local pointed groups on D then 


Fs @ (Re, Ty) = Fo(R., Ty) 0 Fs(R,, Tz) (1.17.1) 
6 


where p and o are respectively the unique local points of R and T on S and ¢' and y’ 
the unique local points of R and T on S ®@D such that 


Brk(p)@ Bra(e)c Bre°M(e’) = and —Brz(o) @Brz(y) < Brz®?(w’) 


Proof. It suffices to prove that F;(R,, 7,) contains F, @,d( Re, Ty’) (cf. [18], Th. 
5.3.). Moreover, if R, < T,, then R c 7 and therefore R, < T,; hence, we may 
assume that |R| = |7| (cf. [17], 2.11). 


The existence of X shows that the canonical image of OR in D, is isomorphic to 
OR and a direct summand of D, as O(R x R)-modules by left and right multiplica- 
tion (cf. the proof of 3.4 in [17]); hence OR is a direct summand of S @,D,, so a 
direct summand of (S @~D,),-, as O(R x R) modules since 


Resp xn(S)@ OR > Resp x a(S) ® Ind jay" (0) > Indi Ry (Ress (Ry (S)) 


where 6(R) is the diagonal subgroup of R x R. In particular, if j’ee’ there are 
b'ej'(S ®_D,)*j’ and an O(R x R)-submodule M of j’(S @ D,) j’ such that 
0 


j(s ® D)j' =M @(Qew-s) 


Let @ be an element of Fs 9 p(R,-, Ty), denote by 6,: RR x T the homo- 
morphism mapping ve R upon (v, g(v)) and choose a’ €(S ®» D)* such that j’* ew’ 
and (v +j’)* =j’* *g(v) for any veR (cf. [17], Prop. 2.12). It is now clear that 
Ind br) is a direct summand of j’(S ®@ D,) j’a’, so a direct summand of S$ @, D, 
as @(R x T)-modules; hence, S° @ Ind bR)(O) is a direct summand of 
S°@,S@~oD always as ©(R*xT)-modules. Consequently, since 
S @@S° = End,(S) as interior P-algebras and 


S° @ Ind§ igh) = Ind igh(Res§.ig)(S°)) 


as O(R x T)-modules, on one hand S ®,@ ss @D is a permutation O(P x P)- 
module and therefore Res} (a)(S°) is a permutation @R-module; on the other hand 


we have 


Res, (Res7(S))° @ Resa(S) = Endo(Res§ ig)(S°)) 


as interior R-algebras (we identify R with 6,(R)). 

Finally, since © can be embedded in End,(Res§(g(S°)) and in 
End,(Res5;py (S°)), it follows that S, and Res,(S,) can be embedded in 
End,(Res§ (g)(S°)) @o Res(S) and therefore S, = Res,(S,) since R has a unique 
local point on that interior R-algebra; hence ge F;(R,, T,) (cf. [17], 2.18). 


1.18. Another consequence of the isomorphism 1.12.1, together with 1.16.1, is that 
the O-algebras A,, A, and 0, L° are pairwise Morita equivalent; in particular, we get 
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evident isomorphisms 
Z(A,) = Z(A,) = Z(0,L°) (1.18.1) 


which induce bijections between me sets of blocks of these 0-algebras. Let a° be . 
block of G lying over b, so that «° = {a°} is a point of G on A,, and denote by a} 
the corresponding primitive idempotent of Z(A, . and by 1(a°) = {r(a°)} the 
corresponding point of L on 0, L°: notice that A ,4° is still an interior P-algebra 
and that we have 


oa $@(O,L°)x(a°) (1.18.2) 


as interior P-algebras. Now it is quite clear from 1.16.3 that, for any local pointed 
group R,o on A,, the following conditions are equivalent 


1.18.3. R,o is a defect pointed group of G,o 
1.18.4. R,o is a maximal local pointed group on A,a 
1.18.5. t(R,o) is a defect pointed group of L,.q) 


and, in that case, an embedded algebra associated with the unique local point of R 
on Resh(S) @o(0,L°) -;e0) (cf. [19], 2.13) and [18], Th. 5.3) is a source algebra of a°: 
in particular, as we say above (cf. 1.1), all the invariants currently associated with a° 
can be computed from Resk(S) @o(0, p° )-(eo) and therefore related to the corres- 
ponding invariants of t(a°) (recall that, by Lemma 5.5 and Proposition 5.15 in [19], 
0,L° can be embedded in the group algebra of a finite central p’-extension L’ of L, 
and therefore t(a°) is a block of L’ too). 


1.19. For instance, we will describe the relationship between the full matrices of 
generalized decomposition numbers of a° and t(a°). Assume that Char(@) = 0 and 
denote by % the quotient field of O, by Irry-(G, a°) the set of irreducible #- 
characters of G associated with a° and, for any yeIrry-(G, a°), by x the corres- 
ponding irreducible #-character of L° (recall that 1.12.1 and 1.16.1 induce a 
Morita equivalence between Aa° and (0, L°)z(a°)). Let R,o be a defect pointed 
group of G,o such that R c Pand m>) + O(cf. 1.16), and Y,o a set of representatives 
Uyo€ R,o for the G-conjugacy classes of local pointed elements on Aa?. Notice that, 
with evident notation, it follows from 1.16.3 that {t(uyc)}y,,eu is a set of 
representatives for the L-conjugacy classes of local pointed elements on 
(0, L°)t(a°). If ye Irr y-(G, a°) and Uyo€ U,o, we denote by x(uyo) and x‘(t(uyo)) the 
corresponding generalized decomposition numbers; moreover, for any ue P, we 
denote by w(u) the “sign” of the trace of the image of u in S* (cf. [18], 1.11.1). Now, 
as in 1.12 of [18], it is not difficult to prove that: 


1.19.1. For any xEltry(G, a°) and any ujoEU,0 we have 
X(Uyo) = @(u)z"(t(Uyo)) « 


1.20. A last remark. Setting G’ = N,(Q;), H’ = Q°Cy(Q), A’ = OG’, B’ = OH’ 
and denoting by b’ the block of H’ such that Br§(b’)Br§(5) = Br§(6) (recall that 
B(Q) = 4C,(Q)), it is clear that b’ has a nilpotent local structure (cf. [18], 1.7). 
Then, with evident notation (cf. 1.6 and 1.7), we have a’ = fp’ = {b’}, N’=G’, 
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N’ = N (cf. 1.7.2) and, for any local pointed group R, on B such that Q; c R,, there 
is a unique local pointed group R, on B’ such that Br§(e’) = Br§(e) (since 
Bro(B’*) = 4Cy(Q)* and B(R) = 4Cy(R)); moreover, it is clear that R, < P,, if 
and only if R, < P, (since both inclusions are equivalent to Rp,s(,) < Pp,8(,))- On 
the other hand, it follows from 1.8.1 that Q = H -q P and from 1.13.1 that 6 is the 
unique local point of Q on B such that Q; c P, (cf. [18], th. 5.3); in particular, if R, is 
a local pointed group on B such that Q; c R, and x an element of N such that 
(R,)* < P,, x normalizes Q;. Consequently, identifying N’ with N through 1.7.2 and 
thus considering N’ as an N-permutation group (since N’ < N), we have 


Ey w(R., Fs) my Ey a(R, Py) (1.20.1) 
for any local pointed group R, on B such that Q; c R, c P,, and therefore 


1.20.2. the group L and the group homomorphisms x: L + N and t: P > L fulfil 
conditions 1.8.1 and 1.8.2 with respect to G’, H’ and b’. 


That is, the group N,(Q;) and the block f=)’ determine L and, since 
B(Q;) = B’(Q;,) as Ng(Q;5)-algebras, they determine the central 4 *-extension L 
and the interior P-algebra o,L° too (cf. 1.15.2); this shows, for instance, that (cf. 
1.12.1 and 1.16.1). 


1.20.3. the O-algebras OGb and ON,(Q;)f are both isomorphic to full matrix 
algebras over 0,L° 


which generalizes the main result of [7]. 

This paper is divided in eight sections. In §2 we list notation, terminology and 
quoted results; although we follow essentially the point of view introduced in [15], 
it is handy to replace interior G-algebras by slightly more general objects named G- 
algebra extensions—which not only simplify statements but suggest the adequate 
equivalence classes of homomorphisms that we may consider—and in §2 we 
develop the corresponding background. In Sect. 3 we study the so-called semi- 
covering exomorphisms: this class of G-algebra exomorphisms has been introduced 
by Puig some time ago to understand the relationship between 0G and 0(G/Z) 
when Z is a central p-subgroup of G*; it turns out that the same class is useful to 
study the relationship between OH and OG when H is a normal subgroup of G such 
that G/H is a p-group, namely to study p-extensions of blocks; actually, together 
with the main result of [18], it is the most important tool to prove Theorem 1.12 
when G/H is a p-group. 

It is not difficult to prove that P/Z(Q) is a Sylow p-subgroup of Ng(Q5)/Cy(Q) 
and that the element of H? (P/Z(Q), Z(Q)) determined by P belongs to the image of 
H? (Ng(Q5)/Cy(Q), Z(Q)) and therefore determines a group extension L of N by Q 
(cf. 1.7.2); but such an approach has the disadvantage that L is not necessarily 
unique up to a unique group exoisomorphism; an important part of this paper is 
devoted to avoid this difficulty, and in Sect. 4 we expose the general criteria we need 


* In order to prove the existence of a bound for the dimension of a block source algebra in terms 
of Cartan integers and dimensions of the irreducible source modules. This result has been 
announced in May 1982, in an Oberwolfach meeting, and will appear in a forthcoming paper 
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to get unique group exomorphisms between group extensions. In Sect. 5 we discuss 
to what extent the local category of B, (cf. 2.8) is determined by N. In §6 we 
essentially handle Theorem 1.12 when N is a p-group. In Sect. 7 we exhibit a wide 
class of conditions which answer the question on the uniqueness of L raised above: 
we have chosen conditions 1.8.1 and 1.8.2 as the easiest ones to state. Finally, in 
Sect. 8 we prove Theorems 1.8 and 1.12. 


2. Notation, terminology and quoted results 


2.1. We follow essentially the notation and terminology introduced in [15], [17], 
[18] and [19] except that we often replace groups by permutation groups and 
interior G-algebras by G-algebra extensions (both terms are defined below). 
Throughout the paper p is a prime number, 4 an algebraically closed field of 
characteristic p and © a complete discrete valuation ring with residue field # (we 
allow the case © = 4). For any A4€0 we denote by / the image of 4 in 4; more 
generally, if M is a torsion free @-module, we denote by X the image of any 
X <M in 4 @@M. Recall that the canonical group homomorphism 0* — £* has 
a unique section (cf. [22], II, Prop. 8); we denote by / the canonical lifting of any 
A€Ek™ to 0”. 


2.2. All the 0-algebras we consider are associative with unity and (-free of finite 
rank as ©-modules. An 0-algebra isomorphic to a finite direct product of full 
matrix algebras over © is shortly called ©-semisimple, and ©-simple if there is just 
one factor. Let A be an (-algebra: all the A-modules we consider are 0-free of finite 
rank; we denote by A° the opposite O-algebra, by A” the group of invertible 
elements of A, by Z(A) the center, by Aut(A) the group of automorphisms, by J (A) 
the Jacobson radical and by A(A) the set of A*-conjugacy classes of primitive 
idempotents — shortly called points — of A. Whereas, if S is a set, S(S) denotes the 
set of all subsets of S to avoid confusion. An ideal a of A means always a two-sided 
ideal, and we set J(a) = a J(A) and denote by A(a) the set of «¢ A(A) such that 
a < a. For any «€ A(A) we denote by A(a) the simple quotient of A associated with 
a, by s,: A—A(a) the canonical homomorphism, by A-a«:A the ideal of A 
generated by a, and by m, the dimension of any simple A(«)-module. 


2.3. A homomorphism f: A > A’ between (-algebras is not required to be unit- 
ary. For any a€A(A) and any a’ € A(A’) we denote by m(f)%. (or mz. when no 
confusion is possible) the dimension of s,-(f(i))* M’ where ica and M’ is a simple 
A’(«’)-module. In non-commutative algebra it is handy to consider homo- 
morphisms up to inner automorphisms: an exomorphism f from A to A’ is the orbit 
ofa homomorphism f: A > A’ in Hom(A, A’) under the evident action of the group 
A* x A’™. Notice that orbits of A* x A’“ and A’* on Hom(A, A’) coincide (cf. 
[19], 2.3), and therefore 


2.3.1. exomorphisms of O-algebras can be composed. 


We denote by Hom(A, A’) the set of these orbits. We say that fe Hom(A, A’) is an 
embedding if Ker(f) = {0} and Im(f) = f(1)A’ f(1). 
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2.4. If Gis a group, we denote by Z(G) the center of G, by D(G) the commutator 
subgroup of G, by O,(G) the subgroup generated by the normal finite p-subgrouips 
of G, by 0?(G) the intersection of the normal subgroups of G such that the quotient 
is a finite p-group, and by N,(H) and C,(H) the normalizer and the centralizer of a 
subgroup H of G. If Gis an extension of G by an (abelian) group Z and y: H > Gis 


a group homomorphism, we denote by Res,(G) the extension of H by Z formed by 
the group G x H of pairs (x, y)eG x H such that w(y) is the image of x in G and 


the group homomorphisms Z-G x H mapping ze€Z upon (z,1) and 


G x H > H mapping (x, y)eG x H upon y; we write Res,,(G) instead of Res,(G) 


when the choice of w is evident from the context. If g:Z—+@©”* is a group 
homomorphism (and G is finite), we may consider the twisted group algebra 
OG/ ¥Y O+(2% — e(2)8) (2.4.1) 
zeEZ,XEG 
that we denote by 0,G when there is no confusion on g, namely if Z = 4* and q is 
the canonical section (cf. 2.1). But in this paper we will systematically consider 
permutation groups rather than groups: if S is a set, an S-permutation group G is a 
group — still denoted by G — endowed with an action on S (i.e. with a group 
homomorphism from G to the symmetric group £(S) of S); then any subgroup of G 
is considered as an S-permutation group with respect to the restricted action. 
Notice that, as any group can be considered as an @-permutation group, 
everything about permutation groups specializes to groups (and then we will omit 
to mention the empty set). 


2.5. Let G and G’ be respectively S- and S’-permutation groups for some sets S$ 
and S’; a homomorphism from G to G’ (or from (G, S) to (G’, S’) to avoid confusion) is 
a pair (y, f) where g: G > G’ is a group homomorphism and f: S > S’ a map such 
that f(x*s) = p(x): f(s) for any xeG and any seS. If S’=S, we denote by 
Hom,(G, G’) the set of S-permutation group homomorphisms from G to G’; similarly, 
we denote by Aut,(G) and Aut,(G’) the groups of S-permutation group auto- 
morphisms of G and G’. The canonical maps G > Aut(G) and G’ > Aut(G’), and 
the structural homomorphisms G — (S$) and G’ > X(S) induce evident group 
homomorphisms G > Aut,(G) and G’ > Aut,(G’); then composition induces an 
action of G x G’ on Hom,(G, G’) and as above we call S-permutation group 
exomorphism from G to G’ any orbit of G x G’ on Hom,(G, G’) and denote by 
Hom,(G, G’) the set of these orbits. Again G x G’ and G’ have the same orbits on 
Hom,(G, G’), and therefore 


2.5.1. exomorphisms of S-permutation groups can be composed. 


Moreover we have an evident map from Hom, (G, G’) to Hom(G, G’) 
(= Homg(G, G’)). 


2.6. Let S be a set and G a finite S-permutation group. A G-algebra B (over () is 
an (-algebra endowed with a group homomorphism g: G > Aut(B); we usually 
write b* instead of p(x~')(b) for any xe€G and any be B. If H is a subgroup of G, 
B* denotes the unitary subalgebra of H-fixed elements of B and, for any subgroup 
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K of H, Tr: BX — B® denotes the relative trace map and B# its image (cf. [19], 
2.8). For any p-subgroup P of G we set B(P) = B’/()'9BG + J(@): B’) where Q 
runs over the set of proper subgroups of P, and we denote by Brp: B” > B(P) (or 
BrB to avoid confusion) the canonical homomorphism (see [19], 2.9 for more 
detail). 


2.7. Let B be a G-algebra. A pointed group H, on B is a pair formed by a subgroup 
H of G (considered as an S-permutation group!) and a point B of B"; we say that B 
is a point of H on B, set B(H,) = B"(B) (cf. 2.2) and denote respectively by Ng(Ag) 
and C,(H,, S) the stabilizers of 8 in N(H) and in the intersection of Cg(H) with 
the kernel of the action of G on S; if H = <x) we say that x, is a pointed element on 
B. If K, is another pointed group on B, we write K, < Hg (or y,€ H, if K = <y)>) 
and say that K, is contained in H, if K < H and for any ie f there is je y such that 
ij = j = ji. A pointed group P, on B(or a point y of P on B) is local if Brp(y) + {0}; 
we denote by #Y,(P) the set of local points of P on B and coherently we set 
P,(H) = P(B") (cf. 2.2). A defect pointed group P, of H, is a maximal local pointed 
group on B such that P, < H, or, equivalently, a minimal pointed group on B such 
that B < TrB(B?-y- B*) (see [19], 2.10 for more detail). Notice that any subgroup 
and any element of G can be respectively considered as a pointed group and a 
pointed element on the trivial G-algebra © (and then we will omit to mention the 
point, which is always {1}). 


2.8. Let H, and K, be pointed groups on B. Any element x of G such that 
(K,)* < H, (where (K,)* = (K*),.) determines an S-permutation group homomor- 
phism from K to H mapping ye K on y* and seS on x" '*s (cf. 2.5); we denote by 
(K,, x, Hg)s the corresponding exomorphism and say that (K,, x, Hg)s is a G- 
exaerghian (or a (G, S)-exomorphism to avoid confusion) from K, to Hg; that is, 
(K,, x, Hg)s is an element of Hom,(K, H) (cf. 2.5) and, if y is another element of G 
such that (K,) < Hg, we have (K,, x, Hg)s = (K,, y, Hg)s if and only if y = zxh for 
some ze Co(Ky S) and some he H (see [12], Ch. I, §1 and [17], Def. 2.1 for similar 
definitions). We denote by Eg 5(K,,H,) the subset of Homs(K, H) of G-exo- 
morphisms from K, to Hg and set Eg s(Hg) = Eg.s(Hg, Hg). Notice that the 
canonical map Fiom,(K, H) > Hom(K, H) (cf. 2.5) induces a surjective map (cf. 
[17], Def. 2.1) 


Eg,s(K,, Hg) > Eg(K,, Hg) (= Eg, g(K,, Hg)) - (2.8.1) 


Moreover, as any composition of G-exomorphisms is a G-exomorphism, Eg s(H,) 
is a subgroup of Auts(H) isomorphic to Ng(H,)/H*Cg(H,,S). We call local 
category of B the category where objects are the local pointed groups on B and 
morphisms are the corresponding G-exomorphisms. If P, is a local pointed group 
on B, we denote by CG(P,, S) the inverse image of O,(Eg_s(P,)) in Ng(P,). 


2.9. An interior G-algebra A (over ©) is an O-algebra endowed with a group 
homomorphism g: G > A”; we usually write x-a-y instead of y(x)ag(y) (see 
[19], 2.11 for more detail). But in this paper it is handy to consider a slightly more 
general structure: a G-algebra extension A is an interior G-algebra — still denoted by 
A-—endowed with a G-stable subalgebra B containing the unity of A. Then B has 
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an evident G-algebra structure; conversely, as in 2.7 of [18], we denote by BG the 
interior G-algebra formed by the free B-module over G endowed with the evident 
product fulfilling 


(bx) (cy) = be*”'xy 


for any x, ye G and b, ce B; except when a different G-stable unitary subalgebra will 
be explicit, we consider BG endowed with Be, where e is the unit element of G, as a 
G-algebra extension. If wy: H > G is an S-permutation group homomorphism, we 
denote by Res,(A) the H-algebra extension defined by the group homomorphism 
gow: H > A* and the same subalgebra B; when H is a subgroup of G and w the 
inclusion map (with ids), we set Res,(A) = Res#(A). Notice that if B = A then we 
have nothing more than the interior G-algebra A and therefore everything about G- 
algebra extensions specializes to interior G-algebras. Similarly, if A = BG then the 
whole structure depends only on the G-algebra B and therefore everything about 
G-algebra extensions specializes to G-algebras by restricting to the structural 
subalgebra; in particular, this remark applies to the exomorphisms defined below. 


2.10. Let A and A’ be G-algebra extensions and denote respectively by B and B’ 
the structural subalgebras. We denote by A @, A’ the G-algebra extension formed 
by the tensor product of A and A’ endowed with the subalgebra B ®, B’ and the 
evident diagonal group homomorphism 


G > A” @(A’)* <(A@A’)* 
© 


A homomorphism f from A to A’ is an @©-algebra homomorphism such that 
{(B) c B’ and f(x-a*y) = x°f(a)- y for any x, ye G and any ae A. We denote by 
Hom(A, A’) the set of G-algebra extension homomorphisms from A to A’. It is clear 
that conjugation induces group homomorphisms from (B°)* to Aut(A) and from 
(B’%)* to Aut(A’), and then composition induces an action of (B°)* x (B’®)* on 
Hom(A, A’): an exomorphism f from A to A’ is the orbit of a homomorphism 
f:A—A’ under this action; again this orbit coincides with the orbit of f under the 
action of (B’°)* and therefore 


2.10.1. exomorphisms of G-algebra extensions can be composed. 


We denote by Hom(A, A’) the set of these orbits. We say that fe Hom(A, A’)is an 
embedding if Ker(f) = {0}, f(A) =f(1)A’f(1) and f(B) = f(1)B’ f (1); notice that 


2.10.2. If f: AA’ is an embedding, for any pointed group H, on B, f(B) is 
contained in a unique point B’ of H on B’, we have f ~'(B’) = B and f induces a &- 
algebra embedding f(H,): B(H,) > B’(H,,). 


Often we denote f and f’ by the same letter. If y: H > G is an S-permutation 
group homomorphism and fe Hom(A, A’), we denote _by Res,( sf’ ) the exomor- 
phism from Res,(A) to Res,(A’) (cf. 2.9) containing f; as above, when H is a 
subgroup of G and y the inclusion map (with ids), we set Res,( f ) = Res$( f x 
notice that (as in 2.12.2 of [19]) 


2.10.3. if f, g¢Hom(A, A’) we have Res, (f) = Res,(g) if and only if f = g. 
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Indeed, if b’¢(B’)* fulfils g(a) = f(a)’ for any ae A then the right multiplica- 
tion by b’ induces a B’G-module isomorphism from B’ f(1) onto B’g(1) since we 
have 

(a’)*b’' =x~'+a' f(1-x)b’ =x7!+a'b'g(1-x) =(a'b’y (2.10.4) 


for any a’€ A’ f (1) and any x €G; it follows that f(1) and g(1) are ( B’®) * -conjugate; 
thus, we may assume that f(1) = g(1) = 1 and then 2.10.4 implies that G fixes b’ (see 
the proof of Lemma 3.7 of [15] for more detail). 


2.11. Let A be a G-algebra extension and denote by B the structural subalgebra. 
An embedded algebra extension (C, g) associated with a pointed group H, on Bisa 
pair formed by an H-algebra extension C and an embedding g: C > Resf(A) such 
that g(1)e f; as in 2.13.1 of [19], it is clear that 


2.11.1. for any pointed group H, on B there is an embedded algebra extension (C, g) 
associated with Hg; moreover, for any embedded algebra extension (C .: 9’) associated 
with H, there is a unique exoisomorphism h: C’ = C such that g’ = go >h. 


Indeed, if je f it suffices to consider the 0-algebra C = j Aj endowed with the 
subalgebra j Bj and the group homomorphism H > C* mapping ye H upon y-j, 
and the embedding g determined by the inclusion C < A. Moreover, if (C’, g’) is 
another embedded algebra extension associated with H,, we may assume that 
g'(1) =j; then g’ induces an H-algebra extension isomorphism C’ = C and the 
uniqueness is clear since we may assume that g’(c’) = h(c’) for any c’e C’. Usually 
we denote by (Ag, f,) an embedded algebra extension associated with H, chosen once 
for ever, and by B, the structural subalgebra. Notice that the argument above, which 
proves the existence and uniqueness of h, shows that 


2.11.2. if Hg is a pointed group on B and g: C > Res#j(A) an H- algebra extension 
RH Hi Bs such that g(1)ef, there is a unique exomorphism h: C= Ag such 


that g = =fyoh 


(see [18], 2.10.1 for the corresponding result on interior G-algebras). Similarly, it is 
not difficult to prove that (cf. 2.7) 


2.11.3. for any pointed groups H, and K,, on B such that K < H, we have K, < H, if 
and only if there is a K-algebra extension 1 exomorphism f°: A, — Resk(Ag ) such that 
f, = Resi (fy) of ®, and then f? is a uniquely determined embedding 


(see [19], 2.13.2 for the corresponding result on interior G-algebras). 


2.12. Let us recall the definition of A-fusions (cf. [17]) — that we only need in 
Sect. 1. Here we assume that S = @ and B = A: that is, G is just a finite group and 
A an interior G-algebra. Let H, and K,, be pointed groups on A; an A-fusion @ from 
K, to Hg is a group exomorphism @: K — H such that @ is injective and there is an 
interior K-algebra embedding f,: A, > Res, (Ag) fulfilling 


Resk(f,) = Resi!( fy) Res¥(f,) . (2.12.1) 


We denote by F,(K,, Hg) the set of A-fusions from K, to Hg and set F,4(H,) = 
F (Hg, Hg) which is a subgroup of Aut(H) (cf. [17], 2.6 and 2.7). We call local fusion 





Extensions of nilpotent blocks 31 


category of A the category where objects are the local pointed groups on A and the 
morphisms are the corresponding A-fusions (cf. [17], Def. 2.15). Moreover, let P, be 
a local pointed group on A and denote by N4~(P) the normalizer in A,“ of the 
structural image of P; the action of N4-(P) on this image induces a group 
homomorphism from F,(P,) to Na: (P)/P+(A})* (cf. [17], Cor. 2.13) and, 
considering N4~(P)/P*(1 + J(A?)) as a central £*-extension of 
N4»(P)/P*(A;)*, we set (cf. [19], 6.7 and 6.14) 


F,(P,) = Resp ¢p \(Ny-(P)/P*(1 + J(A?))) (2.12.2) 


which does not depend on the choice of ft A, > Res#(A) (cf. [19], Prop. 6.8). It is 
not difficult to modify Proposition 6.12 of [19], replacing G by A* and P by its 
structural image P in A%, to prove that: 


2.12.3. If F,(P,) = k”* x F,(P,), there is a group homomorphism 


a: N,.(P,) > A(P,)” 
such that s,(a’) = o(a’) = s,(a)™ for any ae(A”)* and any beN,-(P,). 


By a theorem of Dade (cf. [7], (12) or [16]) this situation occurs in the following 
case: 


2.12.4. If A = End¢(M), where M is an ©G-module, and P stabilizes an O-basis of A 
containing the unity then 2 P,(P) = {y} and F,(P,) = &* x F,(P,). In particular, 
if Q; is a local pointed group on A and for any j € 6 we consider j- M as an © Q-module, 
an injective group exomorphism @: Q — P is an A-fusion from Q, to P, if and only if 
j*M is a direct summand of Res,(M). 


Indeed, the existence and uniqueness of y follow from Corollary 5.8 of [18] and, 
together with Proposition 2.18 of [17], they imply the last statement. 


2.13. Let H be a subgroup of G and C an H-algebra extension, and denote by D 
the structural subalgebra of C; the induced G-algebra extension Ind{(C) is formed 
by the tensor product OG @oy C @oy OG endowed with the distributive product 
fulfilling 


x@®cryx'c’@y’ if yx’eH 
0 otherwise 


(6@c@ Nx @e'@y) =| 


for any x, y, x’, y’€G and any c, c’ EC, with the subalgebra a y®@®D@y ‘and 
with the group homomorphism mapping x €G on 20 xy @1@~y~' where y runs 
over a set of representatives for G/H in G. Moreover, we denote by 


d&(C): C + Res§Ind§(C) (2.13.1) 


the canonical embedding determined by the H-algebra extension homomorphism 
mapping cEC on 1 ®c @ 1 (see [19], 2.14 for the corresponding construction for 
interior G-algebras). Conversely, 


2.13.2. if A is a G-algebra extension and g: C > Res§(A) an H-algebra extension 
embedding such that 1 = Tr$(g(1)), g(1)g(1 =0 for any xeG—H and g(1) 
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centralizes the structural subalgebra B of A, then there is a unique exoisomorphism 
f:Ind§(C) = A such that g = Res$(f)od§(C). 


Indeed, notice first that, if such an exoisomorphism exists, we may assume that 
f(x®@c@®@y)=x°g(c)*y for any x, yeG and any ceC; conversely, it is clear 
that there is an ©-module homomorphism f: Ind§(C)— A mapping x @c@ y 
upon x*g(c)*y and it is easily checked that f is an interior G-algebra iso- 
morphism. Moreover, as g(D) = Bg(1) (cf. 2.10) and B= : Bg(1)*, we get 
B=f() eg ®@D@x"'). Finally, if de(D")* we have Trf(g(d))€(B°)* and. 
(x*g(c)*y)™H9(4) = x+g(c)+y for any x, yeG and any ceC, which proves that f 
does not depend on the choice of g in g. 


2.14. Let A be a G-algebra extension, « a point of G on the structural subalgebra 
B of A, and P, a defect pointed group of G, (cf. 2.7). The following statement gives 
only a weak generalization of the corresponding result on interior G-algebras 
(cf. [15], Prop. 3.6), but it is strong enough to show that the interior G-algebra 
structure of the G-algebra extension A, can “almost” be computed from the interior 
P-algebra structure of the P-algebra extension A, (the “almost” depending on the 
so called multiplicity module of y as explained in Lemma 9.9 of [19] for interior 
G-algebras). 


2.14.1. If Hg is a pointed group on B such that Hy < G, and « < Tr}j(B": B- B") 
then there is a unique interior G-algebra embedding g5: A, > Indj( Ag) such that the 
following diagram of interior H-algebra embeddings commutes 


Res#,(g8) 
Res (A ,) —" 


——+Resf Ind fj( Az) 
i | i dii( Ap) 
As 


In particular, restriction through fy induces an equivalence between the categories of 
A,-and A,-modules. 


Indeed, if i¢a there are j¢B and b’,b” eB" such that ij = j = ji (cf. 2.7) and 
i = Tr§(b'jb”) (since Oi + iJ(B®%)i = iB°i = Tr§(iB"jB"i)); now we may assume 
that A,=iAi and A,=jAj, and that £. Le and KR are determined by 
the inclusions jAj ciAic A (cf. 2.11); then it is easily checked that the 
O-module homomorphism g: A,— Res{Indj(A,) mapping aeA, upon 
et x @jb”*x~'+a-y+b'j@ y~', where x and y run over a set R of representat- 
ives for G/H in G, is an interior G-algebra homomorphism*. Moreover, setting 
c=) cr 1@x°b'j@x' andc” =)... x @jb”*x~' @1, it is clear that c’c’” 
= 1@1@1 and g(a) =c"(1 @a@ 1)c’ for any ae A,; therefore, it follows from 
Lemma 3.7 of [15] that d& (Ag) = Res#(g) off where djj(A,) and fi denote the 
corresponding interior H-algebra embeddings (cf. 2.11.3 and 2.13.1). Finally, the 
fact that g is a uniquely determined embedding (of interior G-algebras) follows from 
2.10.3 and [19], 2.3.4. 


* This homomorphism, which simplifies the proof of Proposition 3.6 of [15], has been pointed 
out by C. Picaronny 





Extensions of nilpotent blocks 33 


2.15. In this paper we are mainly concerned with the following situation: we 
consider a set S, a finite S-permutation group G, a normal subgroup H of G acting 
trivially on S, the G-algebra extension A formed by the O-algebra OG endowed 
with the canonical map from G and with the subalgebra B = OH, and a point « of 
G on B. For our purposes we have to translate some results in Chap. III of [12] to 
the local category of B, (cf. 2.8): we try to be concise rather than complete. 
First of all we translate some terminology. A G,-biexomorphism (or a (G,, S)- 
biexomorphism to avoid confusion) (¢, y) is a pair of elements of Eg 5(Q5, P,) (cf. 
2.8) where Q, is a local pointed group on B such that Q; c G, and P, a defect 
pointed group of G, (cf. 2.7), and we say that | P|/|Q| is the length I(g, w) of (9, w); if 
y =(Q;, x, P,)s and wy =(Q;, y, P,)s, where x, yeG and P, contains both (Q;)* 
and (Q;)’ (cf. 2.8), we set (g, w) = (Q5, x, y, P,)s. Moreover, if ne Eg 5(Q;, P,) we 
write 


(9, /) =(9,n) + (0, W) (2.15.1) 


and for any A4€ Eg 5(Q5,, Qs) and any we Eg 5(P,, P,,.), where Q;, and P’,, are local 
pointed groups on B, we set 


A(g, W)u = (uo pod, powod); (2.15.2) 
notice that Eg 5(Q3,,Q;) +@ implies Q;, c G,, and that Eg ;(P,, P,,) +O ifand 
only if P;, is a defect pointed group of G,. The law of composition 2.15.1. being 
associative (when defined), we say that (@, ) is a linear combination of a set € of 


G,-biexomorphisms if there are nonempty ordered families {A;};-, and {p;} jc) 
of G-exomorphisms and {(9;, ;)};-; of elements of € such that we have 


(9,W) = ps AN Qi, Wi) Mi (2.15.3) 


ie 


in the obvious sense. Then we say that (g, W) is reducible if this G,-biexomorphism 
is a linear combination of the set of all G,-biexomorphisms of length smaller than 
I(@, w), and irreducible otherwise (notice that this terminology differs slightly from 
the terminology introduced in Chap. III of [12]) 


2.16. A generator set of the G,-biexomorphisms is a set © of G,-biexomorphisms 
such that any G,-biexomorphism (gq, ) is a linear combination of €; then it is quite 
clear that (see Lemma 5 of [12], Ch. III). 


2.16.1. if © is a generator set of the G,-biexomorphisms then the set of irreducible 
elements of € is also a generator set. 


To find minimal generator sets of the G,-biexomorphisms we need the following 
fact which generalizes Lemmas 3.9 and 3.10 of [18] and can be proved essentially 
by the same arguments. Notice that for any p-subgroup Q of G we have 
B(Q) = 4C,,(Q) (cf. [19], 2.8.4) and therefore any simple 4C,(Q)-module is 
associated with a local point of Q on B. 


2.16.2. If Q; is a local pointed group on B and K a subgroup of G containing 
Q-C,(Q), there is a unique point v of K on B such that Q; < K,. Then Q; is a defect 
pointed group of K, if and only if Nx(Q;)/Q*Cy(Q) is a p'-group and the simple 
4(Cy(Q)/(H 0 Z(Q)))-module associated with 6 is projective, and in that case Br§(6) 
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is the unique point of B(Q) f where f is the primitive idempotent of Z(B(Q)) such that 
f Br§(5) = Br§(6). In particular, if L is a normal subgroup of K containing Cy(Q) 
and K/L is a p-group, we have K = R«L where R, is a defect pointed group of K,. 


Indeed, setting C = Q-C,(Q)/Q, we have B(Q;)° = 4 and therefore s,(BX) = 
&4 which proves the uniqueness of v. On the other hand, setting N = Nx(Q;)/Q, we 
have B(Q;)" =|N/C|B(Q;)¢ and if B(Q;)¢ = # then the image f of f in Z(4C) 
determines a block of defect zero of C which implies | A(B(Q) f)| = 1. Moreover, 
setting M = R°L, we get easily N,(M) = L*(N,(R,) 0 Nx(M)) (since we have 
R, © M,<.Nx(M), ¢ K,), and therefore Nx(M)/M is both a p-group and a p’- 
group; so, as M is subnormal in K, we get M = K. 


2.17. Let P, be a defect pointed group of G,; as G acts transitively on the set of 
them (cf. [19], 2.10.2), the set of all the G,-biexomorphisms (Q,, 1, x, P,)s is a 
generator set. First of all, we have the following reduction: 


2.17.1. The set of all the G,-biexomorphisms (Q5, 1, n, P,,)s such that ne Ng(Q;) and 
P, contains a defect pointed group of Ng(Qs),, where v is the point fulfilling 
Q; < Ng(Q5),, is a generator set. 


By 2.16.1 it suffices to prove that any G,-biexomorphism (Q;, 1, x, P,)s is a linear 
combination of the union of the set of reducible G,-biexomorphisms and the set 
above; moreover, we may assume that Q + P and then there are local pointed 
groups Q;. and Q5.. on B such that (cf. [15], cor. 1.5) 


Q;22; <P, and (Q;)°3(Q5.CP,. 


Let v be the unique point of N,(Q;) on B such that N,(Q;), contains Q;, Q;, and 
Q;- (cf. 2.16.3), R, a defect pointed group of N,(Q;), containing Qj, (cf. 2.7), y an 
element of G such that (R,)’ ¢ P, (cf. [19], 2.10.2) and n an element of Ng(Q;) such 
that (Q;5..)" < R, (cf. [19], 2.10.2); now it is clear that 


(Q;, 1, x, P,)s = 
(Q;, 1, ys P,)s LE (Q;, ys (Q5))s((Q5)’, 1, n’, P,)s at (Q;, ny, X, P,)s 


and that the G,-biexomorphisms (Q,, 1, y, P,)s and (Q;, ny, x, P,)s are reducible 
(since Q;, + Q; + Q}.) 


2.17.2. If (Qs, 1, x, P,)s is an irreducible G,-biexomorphism then Q, is a defect 
pointed group of Q*C,(Q;,5S),, where gu is the unique point such that 
Q;< Q°Cg(Q;, S),.- 


Indeed, by 2.17.1 we may assume that xe N<(Q;) and that P, contains a defect 
pointed group R, of Q:C,(Q;,S), (notice that Q; — R,); then we have x = mz 
where me N,(R,) and ze C¢(Q,, S) (cf. [19], 2.10.2), and therefore (Q;, 1, x, P,)s = 
(Q;, 1, R.)s(R,, 1, m, P,)s which forces Q; = R,. 


2.18. Let Q; be a local pointed group on B and p the unique point of Q° C,¢(Q;, S) 
on B such that 0; c Q°C,(Q;, S), (cf. 2.16.2). Now we assume that Q; is properly 
contained in P, and a defect pointed group of Q*Cg(Q5, S),. First of all, notice that 
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2.18.1. we have Cp(Q;5,S)< Q. 


Indeed, as H is a G-stable (-basis of B, there is a local point ¢ of Q- Cp(Q;, S) on 
B such that Q-Cp(Q;,S), < P,, and then a local point 6’ of Q on B such that 
0;,< 0:C>,(Q;, S), (cf. [1], Prop. 1.5); but, by 2.16.2 above and Theorem 1.8 of [1], 
we have 6’ = 6 and therefore Q-Cp(Q;, S), < 0° C¢(Q;, S), (cf. 2.16.2); conse- 
quently, Q = Q-C,(Q;,S) (cf. 2.17.2). Recall that Eg 5(Q;) is isomorphic to 
Ng(Q5)/O*C(Q;, 8) (cf. 2.8) and denote by X the image of X < N,(Q;) in 
Eg, s(Qs). 


2.18.2. A G,-biexomorphism (Q5, 1, x, P,)s is reducible if and only if there are p- 


subgroups Ro, ...,R, Of Eg.s(Qs5) such that Np(Q;) < Ro, Np, (Q5) < R, and 
R;-, OR; + {1} fi sign. 


Indeed, if (Q;, 1, x, P,)s is eg there are @o,... , 9, € Eg s(Q5) such that 
Po =(Q5,1, P,)s, On = (Os. x, )s and if 1<ign we have (9;_,,9;)= 
A(Wi-15W; )for some G, pt Doig 1, W;) such that I(W;_,, W;) < |P\/|Q| 
3 2.15); then, by mg 1.5 of [15] and 2.18.1 above, it suffices to set R; = 

(Q;) if @; = (Q5, x;, P,)s, since R; does not depend on the ae of x; (cf. 2.8, 
: 16. 2 and [1], Th. 1.8). Canale assume that there are Ry, ... , R, as in 2.18.2; 
we may assume that Ro= = N,(Q;) and R, = = Np_,(Qs) (Q5);ifO Sign eee by K; 
the inverse image of R; in Ng(Q;), by v; the point of K; on B such that Q; < (K;j),,, 
and by x; an element of G such that ( P,)s' contains a defect pointed group (R;),, of 
(K;),,; notice that K; = R;* Cg(Q5, S) (2.16.2). Now, as we may assume that x» = | 
and x, = x (cf. 2.16.2 and [1], Prop. 1.5 and Th. 1.8), it suffices to prove that 
(Q;, X;- 1, X;, P,)s is reducible for any | < i < n; fix i and let K be the inverse image 
of R;_, A R; in Ng(Q5), v the point of K on B such that Q; c K, and R, a defect 
pointed group of K,; as K, <(K;-,),,_, and K, <(K;),, (cf. 2.16.2), there are 
Z;-1,2;€ Cg(Q5, S) such that (cf. [19], 2.10.2) 


(R,)%- ‘+c (Ri-1)e,_, and (R,)7 & (Ri), 
and therefore we have 


(Q;,x Xji-15 Xi» P,)s = (Qs, 1, R,)s(R,, 2;- 1Xj-15 2jXi» P,)s 5 


but Q; + R, since R;_, 0 R; + {1} (cf. 2.16.2), which completes the proof. 


Then, denoting by M the minimal subgroup of Eg s(Q5) which contains N p(Q;) 
and fulfils the following condition (cf. [12], Ch. II, Prop. 1). 


2.18.3. for any RE Eg 5(Q;) — M the intersection MA M’ is a p’-group. 


It follows from Proposition 2 and Theorem | of Chap. III of [12], and from 
2.18.2 above that (cf. 2.8) 


2.18.4. a G,-biexomorphism (Q5,1,x,P,)s such that xe€Ng¢(Q;) is irreducible 
if and only if x€ Eg 5(Q5) — M, and then we have Q = Po P* and C2(Q;, S) = 
Q- Cg(Q;, S). 


2.19. Finally, assume that H = {1} and that S is endowed with a p’-divisible abelian 
group structure compatible with the action of G; in that case all the points considered 
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above coincide with {1} and we omit to mention them; so, P is just a Sylow p- 
subgroup of G. For any subgroup K of G we denote by H"(K, S) the corresponding 
n-th cohomology group (cf. [4], Ch. XII, §2), and it is quite clear that any G- 
exomorphism geE,5(K,L) induces a group homomorphism H"(g) from 
H"(L, S) to H"(K, S) (cf. [4], Ch. XII, §8). In particular, the inclusion P c G 
induces an injective group homomorphism H"(G, S) — H"(P, S) and its image is 
the intersection of the kernels 


K"((y, W)) = Ker(H"(e) — H"(W)) (2.19.1) 


where {@, w) runs over the set of all the G-biexomorphisms from any subgroup Q of 
P to P (cf. [4], Ch. XII, Th. 10.1). Now, the argument of Lemma 9 of [12] proves 
that 


2.19.2. If X is a subset of G containing N,(P) such that the family 
{(P. 0 P* ',1, x, P)s}xex is a generator set of the G-biexomorphisms, the inclusion 
P c G induces an isomorphism 

H"(G,S) = () K"((Po P* ',1,x, P)s). 


xeX 


3. Semicovering exomorphisms 


3.1. An important tool to prove Theorem 1.12 when G/H is a p-group is a class of 
G-algebra exomorphisms that we name semicovering exomorphisms. This class 
allows us to lift pointed p-groups (see Proposition 3.15 below). Our approach tries 
to make evident the analogy with the covering exomorphisms introduced in Sect. 4 
of [18], the main difference coming from the fact that Theorem 3.13 below fails 
beyond the set of p-subgroups. 


3.2. First of all, it is handy to consider the @-algebra case (i.e. the case where 
G = {1}). Let A and A’ be O-algebras, a and a’ respective ideals of A and A’ (cf. 2.2), 
and f: A > A’ an (-algebra exomorphism (cf. 2.3) such that f(a) < a’. We say that f 
is a semicovering exomorphism from a to a’ if f(J(a)) < J(a’) and f(a) contains a 
maximal commutative @-semisimple subalgebra of a’ (cf. 2.2 and [14], §1) or, 
equivalently, 


m= > m. (3.2.1) 
a €P(a) — A(Ker(f)) a’ eP(a') 


Then we say that - is strict on a if moreover a Ker(f) < J(a) or, equivalently, 
P(a)o A(Ker(f)) = @. Notice that, if f is a semicovering exomorphism from a to 
a’, then f(a) contains all the idempotents of a’ 4 Z(A’). Moreover, it is clear that ri is 
a semicovering exomorphism from a to a’ if and only if 4 @f:# @y~A> 
4 @, A’ is a semicovering exomorphism from 4 to a’ (cf. 2.1), and then fis strict on 
a if and only if 4 @/f is strict on a. When a = A and a’ = A’ we say just that fis a 
(strict) semicovering exomorphism and then f(1) = 1. 


Proposition 3.3. The following conditions on f are equivalent: 


3.3.1. The exomorphism f. A-— A’ is a semicovering from a to a’. 
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3.3.2. We have f(J(a)) < J(a’) and for any «' € Pa’) there is f(a’) < Ala) such 
that m,. = ; pe m, and f(a) < «' for any ae S(a’). 


3.3.3. We have f(J(a)) < J(a’) and there is a map f*: S(P(a’)) > S(AA(a)), 
preserving union and intersection, such that, for any S' < Ala’), we have 
Fina mn, = ci H") m, and + induces an injective unitary exomorphism 
IT A(a)—> [] A’(a’) 
nae wes’ 


In that case, we have yt ha — f*(P(a’)) = P(Ker(f)) and, in particular, fis 
strict on a if and only if f*(P eigad P(a). 


Remark 3.4. If fisa semicovering exomorphism from a to a’, it is clear that 
FHS ') is the disjoint union |_)we¥ S(a’) for any Y’ c P(A’), and that S(a’) is 
the set of «€PA(A) such that m(f)z.+0 (cf. 2.3) for any a’e€VA(a’) (since 
my 2 Vinca) m(f)z-m,). 


Proof. If f is a semicovering exomorphism from a to a’ and T is a maximal 
commutative (-semisimple subalgebra of a, then f(7) is a maximal commutative 
0-semisimple subalgebra of a’ (cf. [14], §1) and, in particular, for any primitive 
idempotent i of T such that f(i) + 0, f(i) is a primitive idempotent of a’; thus, for 
any «’€ P(a’), we have 


=|’ 9 f(T) =YlanTl=hm, 


a a 
where « runs over the set of points of a such that f(a) < «’. 


Assume now that 3.3.2 holds. To prove 3.3.3 it suffices to consider 
ft: S (P(a’)) +S(Al(a)) mapping Y'’c Ala’) upon the disjoint union 
wey F(a’). Indeed, it is clear that such a map preserves union and intersection; 
moreover, we have 

m=) YY m= m, (3.3.5) 
aes’ aeS'’ aEef(a’) ae f*(S’) 
and, in particular, )'x’<a(a’) My = ae faa ym, which proves that A( f~ 1(a’)) — 
f(A a’)) = A(Ker(f)) (since ep Re xe f*( P( a’)) there is a’€P(a’) such that 
f(a) ca’, whereas if aeA(f~' a’)) — f*(P (a’)), it follows from 3.3.5 that 
m(f)z%=0 for any «’€A(A’)); ce as f(J(a)) < J(a’), f induces a unitary 


exomorphism 
I] 4@™—> [] 4 @)-> [] 42) 
aeP(a) a’ eP(a') aes’ 
and the kernel is clearly equal to [], < 44) _ jy) A(@)- 


If 3.3.3 holds and T and 7” are respectively maximal commutative (-semisimple 
subalgebras of a and a’ such that f(T) < 7’, we have 


rank,(7’)= )} m= _) m,Srank,(f(7)), 
a’ eP(a’) aef*(P(a’)) 


and therefore f(T) = 


Proposition 3.5. Let b and b’ be respectively ideals of A and A’ such that f(b) < b’. 
If f is a semicovering exomorphism from a to a’ and from b to b’ then f is 
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also a semicovering exomorphism from a+b to a’ +b’ and the associated map 
f*: S(Al(a’ + b’)) > S(Al(a + b)) extends the associated maps 


S(PA(a’)) + S(A(a)) and S(A(b’)) — S(A(b)) . (3.5.1) 
If moreover f is strict on a and b then tf is also strict on a + b. 


Proof. Since J(a + b) = J(a) + J(b), if F is a semicovering exomorphism from 
a to a’ and from b to b’, we have f(J(a + b)) < J(a’ + b’). On the other hand, 
as P(a’ + b’) = A(a’) UV A(b’), it follows from 3.3.2 that for any «'€A(a’ + b’) 
there is S(a’) < A(a +b) such that m,, = Va ex(a') Ma and f(a) ca’ for any 
ae F(x’). Thus, by Proposition 3.3, fi is a semicovering exomorphism from a + b 
to a’+b’; Then, by Remark 3.4, f*:S(P(a’ + b’)) >S(A(a+b)) maps {x} upon 
S(a’) and therefore extends the associated maps 3.5.1. Finally, the last statement 
follows from the last statement of Proposition 3.3. 


Proposition 3.6. Let A” be an ©-algebra, a” an ideal of A" and g: A’ A” an 
exomorphism such that g(a’) < a”. 


3.6.1. If f and g are respectively semicovering exomorphisms from a to a’ and 
pee" a’ to_a” then g of is a semicovering exomorphism from a to a” and we have 
Gof)* = = f' *og*. In that case, if two semicovering exomorphisms among fg gj and gof 
are respectively strict on a, a’ and a, so is the third. 


3.6.2. If g°f is a semicovering exomorphism from a to a” and g(J(a’)) < J(a”) then g 
is a semicovering from a’ to a”. If moreover g is strict on a’ then f is a semicovering 
from a to a’. 


Proof. If f and g are respectively semicovering exomorphisms from a to a’ and 
from a’ to a”, we have g(f(J(a))) < g(J(a’)) < J(a”) and, for any maximal com- 
mutative @-semisimple subalgebra T of a, f( 7) is a maximal commutative 0- 
semisimple subalgebra of a’, whence g(f(T)) is a maximal commutative 0- 
semisimple subalgebra of a”. Conversely, if g(f(T)) and T’ are respectively 
maximal commutative (-semisimple subalgebras of a” and a’ in such a way that 
f(T) < T’, it is clear that g( T’) = g(f(T)) which proves the first statement in 3.6.2. 
If moreover a’ M Ker(g) < J(a’), the intersection a’ \g~‘(J(a”)) has no nonzero 
idempotents and therefore we get f(J(a)) < a’ Ng~‘(J(a”)) < J(a’); on the other 
hand, as T’o Ker(g) = a iy equality g(T’)=g(f(T)) and the inclusion 
f(T) < T’ imply that f(T 


Corollary 3.7. Let A and A’ be respectively quotient algebras of A and A’ such that f 
induces a homomorphism ££ A-— A’, and denote respectively by a and a’ the images of 


a in A and a’ in A’. If f is a semicovering exomorphism from a to a’ then f is a 
semicovering exomorphism from @ to a’. 


Proof. By 3.6.1, the exomorphism A > A’ induced by fisa semicovering from a 
to a’ and then, by 3.6.2, the exomorphism f is a semicovering from 4 to @’ (since 
J(a) is the image of J(a) in A). 


Proposition 3.8. Let B and B’ be O-algebras, é:B — A and é':B’ > A’ O-algebra 
embeddings and g:B-—B' a unitary exomorphism such that é'°g=f°é. Set 
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b =e '(a)andb’ = e’~'(a’). If f is a semicovering exomorphism from a to a’ then g is 
a semicovering exomorphism from b to b’. If moreover f is strict on a then @ is strict 
on b. 


Proof. Assume that e’°g = fee and choose maximal commutative ©-semisimple 
subalgebras T of a and 7” of a’ such that T centralizes e(1), T’ centralizes e’(1) and 
f(T) < T’ (which is always possible!); then e~'( 7) and e’~'(7’) are respectively 
maximal commutative @-semisimple subalgebras of b and b’, and we have 
g(e~*(T)) < e~1(T"); moreover, it is clear that f(T) = T’ implies g(e~'(T)) = 
e’1(T’). On the other hand, we have e~ '(J(a)) = J(b) and e’~'(J(a’)) = J(b’). 


Example 3.9. Let G and G’ be finite groups and y: G > G’ a group homomorphism 
such that Im(q) is normal in G’ and Ker(g) and Coker(g) are both p-groups, and 
denote by f: OG > OG’ the O-algebra homomorphism induced by g. Then f is a 
strict semicovering exomorphism. Indeed, since Ker(@) is a p-group, we have 
Ker(f) < J(OG) and, as Im(@) is normal in G’, we get f(J(0G)) < J(OG); 
finally, if P’ is a Sylow p-subgroup of G’, Im(f) is P’-stable and we have 
OG =)...» Im(f)u’ which implies that a maximal commutative 0-semisimple 
subalgebra of Im(/) is still maximal in OG’ (cf. [14], Th. 2). 


3.10. Let G bea finite group: we are ready to discuss the G-algebra case. Let B and 
B’ be G-algebras and g: B > B’ a G-algebra exomorphism (cf. 2.9 and 2.10). We say 
that g is a semicovering exomorphism if, for any p-subgroup P of G, the induced 0- 
algebra exomorphism g”: B? — B’ is a semicovering (cf. 3.2), and then we denote 
by 

(g°)*: S(Pg(P)) + S(Ap(P)) (3.10.1) 


the associated map (cf. 3.3.3). If moreover g” is strict or, equivalently, 
(g’)*(Py-(P)) = P,P) for any p-subgroup P of G, we say that g is strict; notice that 
g’ is strict if and only if we have Ker(g”) < J(B”). Recall that if g” is a semicovering 
exomorphism, y’¢ A, (P) and S(y’) is the set of ye A,( P) such that m(g)}, + 0 then 
g’ induces an injective unitary exomorphism (cf. Prop. 3.3 and Remark 3.4) 


g(P,): [] B(P,)>B(P,). (3.10.2) 
yEeS(y') 
First of all, we prove that the relative trace map preserves semicovering 
exomorphisms: this result is the key point to show in Theorem 3.16 below that 
semicovering exomorphisms of G-algebras can be recognized locally. Notice that 
the analogous statement on covering exomorphisms is true and supplies another 
approach to prove Theorem 4.22 of [18]. 


Theorem 3.11. With the notation above, let P and Q be p-subgroups of G such that 
Qc P, b an ideal of B2 and b’ an ideal of B’? such that g(b) < b’. If the O-algebra 
exomorphism g@ is a semicovering exomorphism from b to b’ then g? is a semicovering 
exomorphism from Tr6(b) to Tr6(b’). 


Proof. We may assume that G = P and P + Q. Arguing by induction on|P:Q|, if R 
is a maximal subgroup of P containing Q then g* is a semicovering exomorphism 
from Tr§(b) to Tr§(b’). Consequently, we may assume that | P:Q| = p, so that Q is 
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normal in P; then B2 and B’ are (P/Q)-algebras and g2 is a semicovering 
exomorphism from )\,.,b“ to }\,. pb” (cf. Prop. 3.5). Since Tr6()),,-p 6") = TrG(b) 
and Tr6(>.,.pb”) = Trg(b’), we may assume that Q = {1} and that P has order p 
and stabilizes b and b’. Moreover, we may assume that 0 = & (cf. 3.2). 


Now J(b) and J(b’) are respectively P-stable ideals of B and B’, and we have 
g(J(b)) < J(b’) since the O-algebra exomorphism g is a semicovering from b to b’ 
(cf. 3.2). Set B = B/J(b) and B’ = B’/J(b’) and denote by g: B-— B' the P-algebra 
homomorphism induced by g, and by b and b’ the respective images of b in B and b’ 
in B’; by Corollary 3.7, the 4-algebra exomorphism | 9 is a semicovering from b to b’. 
On the other hand, if the 4-algebra exomorphism g? is a semicovering from Tr? (b) 
to Tr#(b’), the 4-algebra exomorphism from B? to B’? induced by g? is a semi- 
covering from Tr? (b) to Tr?(b’) (cf. 3.6.1), and therefore it follows from 3.6.2 that g? 
is a semicovering exomorphism from Tr*(b) to Tr/(b’) (since J(Tr{(b’)) maps onto 
J(Tr?(b’))). Hence we may assume that J(b) = {0} = J(b’). In that case, b and b’ are 
actually semisimple 4-algebras and therefore we may assume _ that 
b = Band b’ = B’. Moreover, the exomorphism defined by the inclusion g(B) < B’ 
is clearly a semicovering and, since g(B{) = g(B)i, we may assume that g 
is injective. 

In conclusion, we assume that B and B’ are semisimple and that g is an injective 
semicovering exomorphism of 4-algebras, and then it suffices to prove that g’ is a 
semicovering exomorphism from B/ to B’?. Let I be the set of primitive idempo- 
tents of Z(B); since g is an injective semicovering, Bi and g(i)B’g(i) are simple 
algebras for any ie] and therefore g induces a 4-algebra isomorphism 
Bi = g(i)B’g(i). Moreover, by Propositions 3.5 and 3.8, we may assume that P acts 
transitively on J; then, if |J| = 1 or B’ is not simple, g induces an isomorphism 
B~=B' and therefore an isomorphism Bi = Bi’. Otherwise, B’ is simple and 
|I| = p; now, choosing ie], the trace map Tr{ induces a 4-algebra isomorphism 
Bi = B” = BF and, lifting the action of P on B’ to a group homomorphism 
P > (B’)”, we have B’ = Ind? (g(i) B’g(i)) (cf. 2.13.2); in particular, J(B”) = {0} and 
it is easily checked that B’? = g(i)B’g(i) @,4P = B” ®, AP which proves that g” is 
a semicovering exomorphism. 


Corollary 3.12. Let P, Q and R be p-subgroups of G such that R <Q < P. If g@ is a 
semicovering exomorphism from B to BiZ then g? is a semicovering exomorphism 
from BR to BR. 


Proof. This follows from Theorem 3.11 applied to b = BZ and b’ = Bg. 


Now we generalize Propositions 3.6 and 3.8 to the new context. 


Proposition 3.13. With the notation above, let B” be a G-algebra and h: B’ + B” 
a G-algebra exomorphism. 


3.13.1. If G and h are semicovering exomorphisms, then hog is also a semicovering 
exomorphism and we have ((he g)”)* = (g”)*°(h”)* for any p-subgroup P of G. In 
particular, if two semicovering exomorphisms among g, h and he @ are strict, so is the 
third. 
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S432. 5f he °g is a semicovering exomorphism and, for any p-subgroup P of G, 
we have h(J(B’”)) < J(B’”) then h is a semicovering exomorphism. If moreover h is 
strict then g is a semicovering exomorphism too. 


Proof. This follows easily from Proposition 3.6. 


Proposition 3.14. With the notation above, let D and D’ be G-algebras, é: D + B and 
é’: D'  B’ G-algebra embeddings and h: D>D'a unitary G-algebra exomorphism 
such that & oh = go. If g isa _Semicovering exomorphism then his a semicovering 
exomorphism too. If moreover g is strict then h is strict too. 


Proof. This follows easily from Proposition 3.8. 


The following proposition summarizes the lifting features of semicovering 
exomorphisms. 


Proposition 3.15. With the notation above, assume that g is a semicovering 
exomorphism. Let P, and Q;, be respectively pointed p-groups on B and B' fulfilling 
Q c P, and denote by #(65') the set of 6€ A,(Q) such that g(d) < 3’. 


3.15.1. If Q; < P, for some d€ S(65'), there is y'€Ax(P) such that g(y) < y' and 
Qs P,. 


3.15.2. If gy) < y' for some y'€Px(P) then my. = 595) M3- In particular, if 
Q;. < P,, there is 6€ S(5') such that Q; c P,. 


3.15.3. If 5€ S(5') then Q, is local if and only if Q;. is local and, in particular, 
Q, is maximal local on B if Q;. is maximal local on B’. 


Proof. Recall that (cf. [18], 2.3.1) 


Y _ mim(g)s- = m(g)i- = d _ m(g)}-m}, ; (3.15.4) 
5 A(Q) y €Me(P) 


on the other hand, by 3.3.2 we have m(g)3. = 1 or 0 according to whether 6 belongs 
to £(5’) or not, and therefore if Q; < P, for some d€./(5’) we have m(g)j. + 0 
(cf. 3.15.4) which implies m(g)?, + 0 + m}. for some y’ € Yy-(P); hence, in that case, 
we have Q;. c P,. and, again by 3.3.2, g(y) < y’; moreover, if g(y) < y’ for some 
y'€Py(P), y’ is unique such that m(g)?. + 0 and therefore (cf. 3.15.4) 


ms = m(g)}- = mj, . (3.15.5) 
be S(5') 


If 5¢ Y(5’) is not local, there is a proper subgroup R of Q such that 6 < BQ whence 
g(5) < BZ, and g(6)< 6’ forces 5’ < BR; conversely, if 6’< BY we have 
S(5') < P(BY) since g2 is a semicovering exomorphism from Bg to BY 
(cf. Cor. 3.12) and (g2)* extends the associated map from S(A(BP)) to S(A(B)) 
(cf. Prop. 3.5). 

The following theorem may be considered as the main result of this section. It 
supplies “local” criteria to recognize semicovering exomorphisms. For any p- 
subgroup P of G, we denote by g(P): B(P)— B’(P) the 4-algebra exomorphism 
induced by g. 
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Theorem 3.16. With the notation above, the following conditions on g are equivalent 
3.16.1. The G-algebra exomorphism g: B — B’ is a semicovering. 


3.16.2. For any p-subgroup P of G, the 4-algebra exomorphism g(P): B(P) > B’(P) 
is a semicovering. 


3.16.3. For any local pointed group P,, on B’ there is S(y') < AP) such that 
my = >, ey) Mm, and for any ye S(y') we have 


Brp(g(7)) < Brp(y’) and —_—Brp(g( J(B”-y- B”))) < Brp( J(B’’-'- B’*)) . 


3.16.4. For any p-subgroup P of G and any subgroup Q of P, the ©-algebra 
exomorphism g”: B? + B'? is a semicovering from Bf to Bg. 


In that case, g is strict if and only if g(P) is strict for any p-subgroup P of G. 


Proof. It is clear that 3.16.4 implies 3.16.1, and the converse follows from Theorem 
3.11. On the other hand, it follows from Corollary 3.7 (applied to 
£@ g’:4 @,B” + 4 ®,B’ for any p-subgroup P of G) that 3.16.1 implies 3.16.2. 
Moreover, if g is a strict semicovering, for any p-subgroup P of G we have 
(g°)*(Py(P)) = Ay(P) (cf. Prop. 3.3) and, by Proposition 3.15, we get 
(g?)*(LPy(P)) = LA,(P) or, equivalently, g( P)*(A(B'(P))) = A(B(P)), whence 
g(P) is a strict semicovering too (cf. Cor. 3.7); the converse is clear. 


It follows from Proposition 3.3, applied to g(P):B(P)— B’(P) for any p- 
subgroup P of G, that 3.16.2 implies 3.16.3. Finally, assume that 3.16.3 holds and let 
P be a p-subgroup of G; we will prove by induction on |P| that g?:B’ > B’” is a 
semicovering exomorphism. If Q is a proper subgroup of P, it follows from the 
induction hypothesis and Theorem 3.11 that g” is a semicovering exomorphism 
from BO to Bg. So, by Proposition 3.5, g° is a semicovering exomorphism from 
Ker(Br}) to Ker(Br} ) (notice that this is trivially true if P = {1}); in particular, for 
any ye #A,(P) and any y’e A, (P) — LA,(P) we have m(g)}. = 0 (cf. Remark 3.4). 
Moreover, for any y’¢ #FY,(P) there is W(y’) < AP) such that m,, = Z ef(y') My 
and such that for any ye Y(y’) we have 


Brp(g(y)) ¢ Brp(y') and —_Brp(g(J(B”- B”))) < Brp( J(B”-y': B’*)), 


which implies that g(y)cy’ (since m(g)}-=0 for any 6’¢PA(Ker(Br2 ))) 
and therefore that g(J(B’-y-B"))< J(B’?-y’-B’) (since g(J(B’-y-B?)) 
is now contained in B’’-y’-B’’ and maps onto {0} in B(P,)); hence, as 
Jy) (7) By: BY) = Y, ey J (By B®), G? is a semicovering exomorphism 
from })... yj) B”-y: BY to BY’ -y'- B’? (cf. 3.3.2). Finally, if ye 2A (P) — ),, Sy’), 
where y’ runs over 2Y,(P), we have m(g)}, = 0 for any y’€ A,(P), and therefore 
g(y) = {0}; thus g” is also a semicovering exomorphism from B’-y-B? to {0}. 
Consequently, as 


BP= }Y B?-»-B’+Ker(Brf) and B’= }  B?-y'-B? + Ker(Brf), 
y € LP,(P) y' € LA,(P) 


the statement follows now from Proposition 3.5. 
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4. Auxiliary results on group extensions 


4.1. Let L, L’ and F be groups and p: L— F and p’: L’>F surjective group 
homomorphisms; set Z = Ker(p) and Z’ = Ker(p’). Assume that Z and Z’ are 
p’-divisible abelian groups, so that they become Z,,) F-modules through the re- 
spective actions of L and L’. Moreover, assume that L and therefore F and Z are 
finite, denote by P a Sylow p-subgroup of L, so that Z c P, and set p(P) = P. As 
usual, we consider respectively L and L’ endowed with p and p’ as group extensions 
of F by the Z,,,F-modules Z and Z’. 

4.2. It is well-known from homological algebra (cf. [4], Ch. XIV, Th. 3.1 and 4.2) 
that L and L’ determine respectively elements he H?(F, Z) and h'eH?(F, Z’) (cf. 
2.19), and that there is a group homomorphism o: L - L’ lifting id; and extending 
a Z,,,F-module homomorphisia 6: Z— Z' if and only if the homomorphism 
H?(F,@) maps h upon h’. Moreover, as restriction induces an injective map 
H?(F, Z) + H?(P, Z) (cf. 2.19), there is such a group homomorphism o: L > L’ if 
and only if there is a group homomorphism t: P > L’ lifting the inclusion P c F 
and extending 0. But, in this case, o does not necessarily extend t or, more generally, 
any L’-conjugate of t; indeed, the “difference” between t and the restriction of o to 
P determines an element of H'(P, Z’) (see the proof of Proposition 4.4 below) 
which is not necessarily the zero element. In this section we discuss a sufficient 
group-theoretical condition to guarantee that o extends t up to L’-conjugation, 
which is still useful for non-abelian extensions (see Proposition 4.9 below). 


4.3. We consider F endowed with the group homomorphism F — Aut(Z’) in- 
duced by p’ as a Z’-permutation group (cf. 2.4). Recall that, considering the trivial 
F-algebra © (and omitting to mention the points), an (F, Z’)-biexomorphism 
(p, W) = (R, x, J, P)z (cf. 2.15) is a pair of elements of E, z-(R, P) (cf. 2.8) where R is 
a p-subgroup of F and x, ye F fulfil R* c P and R’ c P; moreover, a generator set 
of the (F, Z')-biexomorphisms is a set € of (F, Z’)-biexomorphisms such that any 
(F, Z')-biexomorphism is a linear combination of € (cf. 2.15.3 and 2.16). Notice 
that, by 2.16.1, 2.17.1 and 2.18.4, the family {(P 7 P* ', 1, x, P)z-}- ., is a generator 
set of the (F, Z’)-biexomorphisms. 


Proposition 4.4. Let X be a subset of F containing N,(P) such that the family 
{(P. 0 P* ',1, x, P)z-}z-x% is a generator set of the (F,Z’)-biexomorphisms. Let 
t:P —L’ be a group homomorphism fulfilling the following conditions: 


4.4.1. We have p'(t(u)) = p(u) for any ue P, and t(z*) = t(z)* for any ze Z and any 
xeF. 


4.4.2. If X€X there are yeL and y'eL’ such that p(y) = zx = p'(y’) for some 
zZeEC(P A alte Z’) and t(u”) = t(u)” for any uep (Po P*'). 


Then there is a unique group exomorphism 6:L— L' having a representative o 
such that p’°o = p and o(u) = t(u) for any ue P. 


Remark 4.5. Conditions 4.4.1 and 4.4.2 are obviously necessary. 
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Remark 4.6. Notice that if g, ye Hom(L, L’) fulfil the equalities p’° @ = p’°w and 
g(u) = w(u) for any ue P then there is z’€ C,.(y(P)) such that w(x) = g(x)’ for 
any xeéL. Indeed, the 1-cocycle mapping xe L upon (x) @(x)eZ’ determines 
the zero element of H'(L, Res,.,(Z’)) since its restriction to P is the trivial 
homomorphism from P to Z’ (cf. 2.19); hence there is z’€ Z’ such that 


W(x)” * p(x) = 2’ *2* = 2’ * p(x)" *2’g(x) 


for any xe L, whence (x) = g(x)’. 


Proof. Denoting by 6:Z — Z’ the Z,,) F-module homomorphism induced by t and 
arguing as in 4.2, we get a group homomorphism 9: L > L’ such that p’> g = p and 
y(z) = t(z) for any zeé Z; in particular, we have p’(g(u)) = p(u) = p’(t(u)) for any 
uéP, and therefore t(u) = p(u){(u) where the map ¢: PZ’ is a 1-cocycle; 
moreover, as {(z) = 1 for any ze€ Z, ¢ induces 1-cocycle £: P > Z’ which determines 
an element h of H!(P, Z’). Now it suffices to prove that h is the restriction of an 
element of H'(F, Z’); indeed, in this case it is quite clear that there is a 1-cocycle 
&:F + Z’ extending (, and then it is easily checked that the map a: L > L’ assigning 
to xe L the product o(x)é( p(x))eL’ is a group homomorphism extending t and 
fulfilling p’°o = p. Moreover, the uniqueness of ¢ follows from Remark 4.6. 


To show that h belongs to the image of H'(F, Z’), it suffices to prove that, 
for any xeX, the l-cocycle PoAP*'—+Z' mapping wePOP*' upon 
f(a)" f(a) ‘eZ’ is a 1-coboundary (cf. 2.19.2). By condition 4.4.2, for any 
xeX there are yeL and y’eL’ fulfilling p(y)=zx=p'(y’) for some 
ze€Cp(PAP* ',Z’) and t(u”) =t(u)” for any uep '(PAP* '); now, setting 
z’ = w(y)y’*, we have 


p(u)l(u) = t(u) = (wy = e(u?)” wy" = eu) Cw)” 
and therefore 
C(u)~ '$@*)F* = E(u) 16 (u?)”"' = 2’ 12/4 


, where ue P lifts u. Consequently, the 1-cocycle above is indeed 


for any ue PA PX 
a 1-coboundary. 
In particular, Proposition 4.4 implies the following splitting criterion. 


Corollary 4.7. With the notation of Proposition 4.4., let t: P-L’ be a group 
homomorphism fulfilling the following conditions: 


4.7.1. We have p'(t(u)) = u for any ue P. 


4.7.2. If X€X there is x'eL’ such that p'(x')= x and 7(u*) = 7(u)* for any 
ie Po P*. 


Then there is a unique group exomorphism G: F +L having a representative 6 such 
that p’°o = idy and a(u) = t(u) for any ue P. 


Proof. This follows trivially from Proposition 4.4 setting L = F and p = id,. 


4.8. Let E be a finite group and 2: L-E and 7’: L’-E surjective group 
homomorphisms. Assume that x and 7’ have the same kernel Q and induce the 
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same group exomorphism from E to Aut(Q); that is, denoting respectively by y and 
yn’ the group homomorphisms from L and L’ to Aut(Q) induced by conjugation, 
and by 7 and n’ the corresponding homomorphisms from E to Aut(Q), we assume 
that 7 = 7. Moreover, assume that Q is a p-group, so that Q c P, and consider L as 
an @ -permutation group (cf. 2.4) (omitting to mention the empty set). 


Proposition 4.9. Let X be a subset of L containing N,(P) such that the family 
{(P 0 P* ', 1, x, P)}xex is a generator set of the L-biexomorphisms. Let t: P > L' be 
an injective group homomorphism fulfilling the following conditions: 


4.9.1. We have n'(t(u)) = n(u) for any ue P, and n(e) = 7’ (e)* for any ee E, where 
To is the automorphism of Q induced by t. 


4.9.2. IfxeX there is x'€L' such that n'(x') = n(zx) for some z€C,(P 7 P* ') and 
t(u*) = t(u)* for any ue PA P* '. 


Then there is a unique group exoisomorphisma: L = L’ having a representative o 
such that n'°o = nm and o(u) = t(u), for any ue P. 


Remark 4.10. Conditions 4.9.1 and 4.9.2 are obviously necessary. 


Remark 4.11. Considering respectively L and L’ endowed with z and 2’, and with 
left multiplication as E-permutation groups (cf. 2.4), condition 4.9.1 implies that t 
induces an evident bijection tg: Hom,(R, P) = Hom,(t(R), t(P)) for any sub- 
group R of P, and then condition 4.9.2 can be restated as follows: 


4.11.1. IfxeX and R = PA P* '," there is pEE,, -(R, P) lifting (R, x, P)€ E,(R, P) 
such that te(p)é E,., ¢(t(R), t(P)). 


Indeed, if 4.9.2 holds it suffices to set p = (R, zx, P),. Conversely, if @ = (R, y, P)¢ 
lifts (R,x,P) we have y=zxv for some zeEC,(R) and veP (cf. 2.8). and if 
Ttr(p)€E,.. -(t(R), t(P)) there is y’eL’ such that 2’'(y’) = x(y) and t(u”) = t(u)” 
for any ue P (cf. 2.8), whence 2’(y’t(v)~!) = x(zx) and t(u*) = t(u)”*®* for any 
ue P. 


Proof. We may assume that F above is the pull-back of 4: E — Aut(Q) and 
the canonical homomorphism Aut(Q)— Aut(Q) (i.e. the subgroup of pairs 
(e, 0)E E x Aut(Q) such that n(e) = 0), and that p and p’ are respectively the 
group homomorphisms mapping xe€Z upon (x(x),n(x)) and x’eL’ upon 
(2' (x’), n'(x’)). Then, it is clear that Z = Z(Q) = Z’; moreover, notice that p and 
p’ induce the same group homomorphism F —> Aut(Z) since nix) = nx’), for any 
xéL and any x’eL’ such that x(x) = 2’(x’), by condition 4.9.1. 


Now, considering F endowed with this group homomorphism as a Z-permut- 
ation group (cf. 2.4), it is still clear that p induces a surjective map 
E,(R, P) > Er, z( p(R), p(P)) for any p-subgroup R of L containing Q; in particu- 
lar, setting X = p(X) and P= p(P), the family {(P 7 P* ', 1, x, P)z}--z is a 
generator set of the (F, Z)-biexomorphisms (since any (F, Z)-biexomorphism can 
be lifted to an L-biexomorphism). Therefore it suffices to prove that conditions 
4.4.1 and 4.4.2 (with respect to X and 1) follow from conditions 4.9.1 and 4.9.2. 
Indeed, it follows then from Proposition 4.4 that there is a group homomorphism 
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o:L — L' which extends t and fulfils p’° o = p, whence x’° o = 1; moreover, o is an 
isomorphism since t is injective and |L| = |L’|, and the uniqueness of ¢ follows 
from Remark 4.12 below (or directly from Proposition 4.4). 

First of all, we prove that 4.9.1 implies 4.4.1; as (y’(t(u))°)(v) = vo" = n(u)(v) for 
any ue P and any ve Q, it is clear that 2’(t(u)) = 2(u) implies p’(t(u)) = p(u) for 
any ueéP; moreover, if xeL and x’eL’ fulfil p(x) = p’(x’) then t(n(x)(z)) = 
n'(x’)(t(z)), whence t(z*) = t(z)* for any ze Z. Secondly, by condition 4.9.2, if 
xeX and xeX lifts x there is x’eL’ such that 2’(x’)=2(zx) for some 
zeC,(POP* ’) and t(u**) = t(u)* for any ue Po P* ', and it suffices to set 

=zx and y’=x’ in 4.4.2; indeed, as t(n(y)(v)) =n'(y’)(t(v)) for any veQ 
(since Qc PA P* '), we have p'(y’) = p(y) =zx where Z= p(z) belongs to 
Cr(P 0 PX ',Z) (once again since Qc Po P* '), and t(u”)=1(u)” for any 
uePoP*' =p '(PoP*'). 


Remark 4.12. If g and are injective group homomorphisms from L to L’ fulfilling 
the equalities z’° g = n'°w and g(u) = w(u) for any ue P, there is z€ Cg(P) such 
that w(x) = g(x") for any xe L. Indeed, with the notation above where t is the 
restriction of g and yw to P) it is easily checked that p’° y = p’°y, and it suffices to 
apply Remark 4.6. 


5. On the local structure of block extensions 


5.1. Let G be a finite group and H a normal subgroup of G, set A = OG and 
B = @H, and consider A endowed with B as a G-algebra extension (cf. 2.9). Let « 


and £ be respectively points of G and H on B (cf. 2.7) such that Hy c G,, set 
N = N,(Hg) and N = N/H, and denote by X the image of X < N in N. In this 
section we will discuss to what extent the local category of B, (cf. 2.8 and 2.11) is 
determined by the quotient N. Actually we are only concerned with the case where 
the local structure of H, is nilpotent (cf. [18], 1.7) - what we assume from 5.6 on — 
which is completely handled in Theorem 5.8 below. However it is not difficult to 
discuss the general situation from our results, as we explain in Remark 5.9 below. 


5.2. Obviously £ has just one element b and « = {Try (b) }; hence f is still a point 
of N on B, therefore we get A, = Ind§ n(Ag) (cf. 2.13.2); in particular, it is easily 
checked that the inclusion N c -G induces an equivalence between the local 
categories of B, and B, (cf. 2.8). That is, for our purposes it suffices to consider the 
local pointed groups on B contained in Ng, and the next statements show that the 
set of their images in N is the full set of p-subgroups. 


Proposition 5.3. A local pointed group P, on B such that P, < Ng is a defect pointed 
group of G, if and only if P is a Sylow p- subgroup of N and there is a defect pointed 
group Qs of Hz such that Q; < P,, and then we have Q = Ho P. 


Remark 5.4. Notice that Proposition 5.3 depends only on the G-algebra structure 
of B and therefore applies to each subgroup of G containing H. 


Proof. Assume first that P, is a defect pointed group of G,; in particular, P, is also a 
defect pointed group of N,, and therefore a defect pointed group Q, of H, has an 
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N-conjugate contained in P,; so we may assume that Q; c P,. Moreover, as H 
is a P-stable O-basis of B, there is e¢€ YA,(PH) such that (POH), c P, 
(cf. [1], Prop. 1.5); hence (PO H),< Ng and therefore (POH), < Hg (since 
s,(b) + 0); thus there is he H such that (P 7 H), < (Q;)" which forces PA H = Q 
(since Q < PH). On the other hand, f is still a point of P- H on Band P, a defect 
pointed group of P- ‘Hg; th therefore, as Ng(P,) < N (since by = y), the Frattini 
argument shows that N,¢(P,) = N n(P), so that N,(P, )/ P+ C,(P) maps surjectively 
on Nx(P)/P; but the quotient N,(P,)/P* Cy(P) is a p’-group (cf. 2.16.2); hence Pis 
a Sylow p-subgroup of N. 


Conversely, assume that P is a Sylow p-subgroup of N and that there is a defect 
pointed group Q, of H, such that Q; < P,; now, if R, is a defect pointed group of N, 
(and therefore of G,) such that P, c R,, we have P = R and, by the argument 
above, Q = ROH so that PO H = Ro GH; it follows that P = R, and so P, = R,. 


Proposition 5.5 Let P, and Q; be respectively defect pointed groups of G, and H, 
such that Q;< P,. For any subgroup R of P containing Q there is a unique 
e€ LP,(R) such that R, < P,, and then it fulfils the following conditions: 


5.5.1. We have Q; — R,, Ng(R.) < Ng(Qs) and Ng(R,) 0 Cu(Q) = Z(Q):Cu(R). 


5.5.2. Let L, be a pointed group on B such that R, < L, and Cy(R) < LC N,(R,). 
A local pointed group T, on B such that R, < T, < L, is a defect pointed group of L, 
if and only if T is a Sylow p-subgroup of L. 


Proof. First of all, notice that P, < N,; indeed, anyway there is xeG such that 


(P,)* < Ng (since a defect pointed group of N, is a defect pointed group of G,), and 
then there is he H such that (Q;)" = Q, (since (Q;)* < Hz), whence xeéN (since 
Ng(Q5) < N). Moreover, as H is a P-stable O-basis of B, there is e¢ #A,(R) such 
that R, c P,, and if e’¢ YA,(R) there is also 6’ YP,(Q) such that Q; c R,. 
(cf. [1], Prop. 1.5); now, assuming that R, < P, we get Q; < N, and therefore 
Qs < H, (since bd’ = 0’). 


Let L, and T, be pointed groups on B such that T, is local, R, c T, c L, 
and C,(R) c LC N,(R,), and set A = Brg(A), W = Breg(w) and é = Bra(e). As H 
is an L-stable O-basis of B, it is easily checked that Brpg(B’) = B(R)* and 
Br,(B") = B(R)’, and therefore A, w and é are respectively points of L, T and R on 
B(R) fulfilling R-< Tj < Lz; moreover, as B(R) = 4Cy(R) (cf. [19], 2.9.2) and 
&e P(B(R)), there i is a block f of C,(R) such that 2 = { f} where f = Brg(f) (since 
fe =é& and L normalizes R,), and therefore 1 is still a point of C,(R), 
N,(R,) 0 Cy(Q) and R-Cy(R) on B(R). 

First of all, it follows from Proposition 5.3 applied to Lj, Cy(R)z and 7; that if 
T; is a defect pointed group of L; then T is a Sylow p-subgroup of L (and it is clear 
that if 7, is a defect pointed group of L, then 7; is a defect pointed group of Lj); 
conversely, again by Proposition 5.3, if T is a Sylow p-subgroup of L then T, isa 
defect pointed group of L-H, (cf. Remark 5.4) since Q;.c T, for some 
5"e LP,(Q) (and then Q,. is a defect pointed group of H,), and therefore, as 
L, < L-H,, T, is a fortiori a defect pointed group of L,. In particular, Rzis a defect 





2s B. Kiilshammer and L. Puig 


pointed group of R* C,,(R)j, and therefore the block f of R*C,,(R) has a nilpotent 
local structure (cf. [18], 1.7); but, as fe = é and P,, contains both R, and R,,, we have 
also fe’ =& where é = Bra(e’) (cf. [1], Th.1.8); consequently, we have é = @’ 
(cf. [2], Th. 1.2 or [18], Th. 1.6), whence ¢ = e’. It is now clear that 6’ = 6 (since 
Q; < P,) and that Ng(R,) < Ng(Q;5) (since Q = ROH). 

Finally, set B = Brg(f); as Brp(BR#) = B(R)Ne-#®) (cf. [1], Lemma 1.12°) 
and R, is a defect pointed group of R* H, (cf. Prop. 5.3 and Remark 5.4), B is a 
point of Nz.q(R) on B(R) and R; is a defect pointed group of Nr.y(R)j. On the 
other hand, é is also a local point of Z(Q) on B(R) since é € A(B(R)), and setting 
M =N,(R,) 1 Cy(Q) we have Z(Q);< Mj since fe = & Now, as Z(Q)-;c Rz 
M;z¢ Na.x(R)z and RAM = Z(Q), it is quite clear that Z(Q); is a (the unique) 
defect pointed group of Mj, and it follows from Proposition 5.3 (applied to 
M;, C,(R)z and Z(Q);) that M/Z(Q)* C,(R) is a p’-group; but for any p’-element s 
of M we have [s,R] “RO H=Q and [s,Q]=1, whence s centralizes R 
(cf. [9], K .I, 4.4); so M/Z(Q)*Cy(R) = {1}. 


5.6. From now on we assume that b is a block of H with nilpotent local structure 
(cf. [18], 1.7). Let P, and Q; be respectively defect pointed groups of G, and H, 
such that Q; c P,, and R, a local pointed group on B such that Q; c R, c P,. As 
Ny(Q5) = 2° Cy(Q) (cf. [18], 1.7.2), on one hand it follows from 5.5.1 that 


Nu(R,) = Q*Cy(R) . (5.6.1) 


On the other hand, as N = H:N,(Q;) by the Frattini argument, the inclusion 
Ng(Q5) < N induces a group isomorphism N ~ N,(Q;)/Q*Cy(Q) (cf. 1.7.2); then 
this isomorphism and the group homomorphism yn: N,¢(Q;) > Aut(Q) induced by 
conjugation determine a group homomorphism 


n:N — Aut(Q) (5.6.2) 


and we set N = N/Ker(n) and consider N, endowed with the action on N induced 
by left multiplication, as an N-permutation group (cf. 2.4); notice that_the group 
exomorphism 7 does not depend on the choice of 6, and therefore N does not 
depend on the choice of Q;. 


5.7. Now consider the trivial N-algebra © (omitting to mention the points) and 
the full subcategory of the local category of the N-algebra B, (cf. 2.8) formed by the 
local pointed groups on B which contain a defect pointed group of H,; the 
canonical homomorphism N > N induces an evident functor from this sub- 
category to the local category of the (N, N )-algebra © (cf. 2.6, 2.7 and 2.8); precisely, 
for any local pointed group 7, on B such that Q; < T, < P,, we have the map 


Ey(R,, Ty) > Ex,x(R, T) (5.7.1) 


assigning to (R,, x, T,)€Ey(R,, T,) the (N, N )-exomorphism (R, x, T)z (cf. 2.8). 
Recall that we denote respectively by CX(R,) and Ch (R, N ) the inverse images of 


© Notice that, in Lemma 1.12 of [1], the hypothesis on P is irrelevant 
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O,(Ey(R,)) in Ny(R,) and O,(Ey. #(R)) in N(R) (ef. 2.8), and that there is a unique 
point uw of C&(R,) on B such that R, < CK(R,), (cf. 2.16.2). 


Theorem 5.8. With the hypothesis and notation above: 


5.8.1. We have O?(Cy(R,)) = O?(Ca(R, N)) and the maps 5.7.1 are surjective. 


5.8.2. The pointed group R, is a defect pointed group of C(R,), if and only if the p- 
group R is a Sylow p- aabareip of CK(R, N ), and then the maps 5.7.1 are bijective. 


Remark 5.9. Notice that, without any hypothesis on b, the unique point / of 
Q°C,(Q) on B such that Q;< Q-C,(Q), (cf. 2.16.2) determines a block of 
Q-C,(Q) with nilpotent local structure (cf. [18], 1.7) and therefore Theorem 5.8 
always applies to the group N,(Q;) and the normal subgroup Q°C,(Q). 


Proof. As Cy(R,) < Cx(R, N), we have already 0°(C,(R,)) ¢ O?(Cx(R, N)). If 
xéN is such that R* c T we have (R: H)* < T:H where xe Ng (Q,) lifts x; but is 
still a point of both R-H and 7-H on B, and therefore (R-H,)* < T:H,; 
moreover, it follows from Proposition 5.3 applied to both R:H and T°-H (cf. 
Remark 5.4) that R, and T, are respectively defect pointed groups of R: H, and 
T+ Hg; hence we may choose the element x fulfilling (R,)* < T,, which proves 
that the maps 5.7.1 are surjective. Moreover, if x is a p’-element of C a(R, N), we 
may assume that x is also a p’-element which normalizes R, (arguing as above and 
eventually replacing x by a suitable power of x); then, as xe Ker(m) (cf. 5.6.2), x 
centralizes Q and, since [x,R] c HOR =Q (cf. prop. 5.3), it follows that x 
centralizes R (cf. [9], K .I, 4.4), whence x € O?(C,(R,)). 

In particular, we have actually proved that Ny(R,) maps onto N(R) and that 
the kernel of the induced group homomorphism 


Ey(R,) > Ex,x(R) 


is a p-group, whence we get Ch(R,) = Cr r(R, N ); hence, it follows from 5.5.2 
that R is a Sylow p-subgroup of C%(R, N) if and only if R, is a defect pointed 
group of Ci(R,),. In that case, if (R,,x,7T,) and (R,, y, 1) are two N-exo- 
morphisms such that (R, x, T)x = (R, y, T)i, we have y = zxu where ueT and 
zER-Cy(R,) since R- Cx(R, N) = Ch(R, N) = CR(R,) = R°Cy(R,) (cf. 5.5.2); in 
particular, we get R’ =(R°H)!)OT=(R°‘H)“AT=R™ since TAH=Q 
(cf. prop. 5.3); hence, as T, contains both (R,)” and (R,)™, the difference yu~* x7! 
belongs to Ng(R,) 0 R* Cy(R,)° H (cf. Prop. 5.3), and therefore we have (R,, x, T,) 
= (R,, y, T,) by 5.6.1. 





5.10. If xeN¢(Q;) we know by Proposition 5.5 that there is a unique 
ye LP,(P oO P*) such that (P 4 P*), < P, and we denote the local pointed group 
(PO P*), by P,a(P,)*. Although this notation is not necessarily symmetric, it 
becomes symmetric if x normalizes P, > (P,)* and then this local pointed group is 
the biggest containing Q; and contained in both P, and (P,)*. In the sequel we 
employ this notation only in the symmetric case. Notice that the family 
{(P, 0(P,)*, 1, x, P,)} where x runs over the set of elements of Ng(Q;) such that 
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P,a(P,)* =(P,0(P,)*)* is a generator set of the N,-biexomorphisms 
(cf. 2.16.1, 2.17.1, 2.18.4 and Lemma 5.12). 


Corollary 5.11. With the hypothesis and notation above, let X be a subset of Ng(Q5) 
such that P, A(P,)* =(P,0(P,)*)* for any xe X. 


5.11.1. If xe X then (P,(P,)*, 1, x, P,) is an irreducible N,-biexomorphism if and 
only if (P © P*, 1, x, P)g is an irreducible (N, N)-biexomorphism, and then we have 
POP* =PaP*. 

5.11.2. The family € = {(P,0(P,)*, 1, x, P,)}xex is a generator set of the Nog- 


biexomorphisms if and only if the family € = {(Pa P*, 1, x, P)z}<z-z is a generator 
set of the (N, N )-biexomorphisms. 


Proof. Let x¢ X and assume that R, = P, 0(P,)”; it follows from 2.17.2, 2.18.4 and 
5.8.2 that to prove 5.11.1 we may assume that R, is a defect pointed group of 
CK(R,), (cf. 5.7); then, by 5.8.2 we have Ey(R,) = Ex, %(R) and statement 5.11.1 
follows from 2.18.4. 


On the other hand, it follows from 2.16.1 and 5.11.1 that to prove 5.11.2 we may 
assume that all the elements of € and © are irreducible, and in particular that 
Po P* = Po P* for any xe X, since 


(P70 P*, 1, x, P)z =(P 7 P*, 1, PA P* ')3(Po P* ',1,x, P)z. 


As P is a Sylow p-subgroup of N (cf. Prop. 5.3) and the maps 5.7.1 are surjective 
(cf. 5.8.1), it is quite clear that if € is a generator set of the N,-biexomorphisms then 
Cisa generator set of the (N, N )-biexomorphisms. Conversely, assume that Cisa 
generator set; by 2.17.2, 2.18.4 and Lemma 5.12 below, it suffices to show that an 
N,-biexomorphism (R,, 1, x, P,) is a linear combination of € (cf. 2.15.3) assuming 
that R, is a defect pointed group of Cf(R,), (cf. 5.7); but, by hypothesis, the (N, N )- 
biexomorphiem (R, 1, x, P)s i is a linear combination of ©, and therefore there are 
ordered families {€;};-; and {C;};<, of (N, N )-exomorphisms and { X;};-; of elements 
of X such that (cf. 2.15.3) 


(R, 1, x, P)g = ¥ E(P 7 P*, 1, X, P) gl; 
iel 


now, choosing liftings x;¢ X of x;, ¢;¢ Ey(R,, P, 0 (P,)*) of é,, and f,€ Ey(P,, P,) of 
¢; (cf. 5.8.1), it follows from 5.8.2 that we have 


(R,, 1, x, P,) me > o(P, O(P,)™, 1, Xis P,)C; > 


Lemma 5.12. With the hypothesis and notation above, let T, be a local pointed group 
on B such that T, < P, and v the point of Ck(T,) on B such that T, < CR(T,),. If T, 
is a defect pointed group of Ck(T,), then Q; < Ty, 


Proof. It suffices to prove that Ny(T,)/Cy(T) is a p-group; indeed, as N,,(7,) is 
normal in Ny(T,), in that case we have Ny(T,)< Ch(T,) and a fortiori 
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Ny.y(T,) < CK(T,); thus, if 4 is the point of N;.,(7,) on B such that 
Ty < Nyp.y(Ty), < Ch(Ty), (cf. 2.16.2), then Ty is still a defect pointed group of 
N7.y(T,),; but, since T, < P, < Ng, we have by = and so bj = A, whence 
N7.y(T,y), < T* Hg; consequently, T, is also a defect pointed group of T-H, 
(cf. [15], Cor. 1.4), whence Q; < T, by Propositions 5.3 and 5.5. 


Set U = TO H; as H is a P-stable (-basis of B, there is pe #P,(U) such that 
U,< T, (cf. [1], Prop. 1.5); thus, as U, < T, < P, < Ng, we have U, < H, (since 
bo = 9); therefore, E,(U,) is a p-group (cf. [18], 1.7.2) and the block of C,(U) 
associated with g is still nilpotent (cf. [2], Th.1.2), which implies that 
Nx(T,) < Ny(U,) (cf. [1], Th. 1.8 and [2], Th. 1.2). So, for any p’-element s of 
N,(T,), we have [s, T] c HAT =U and seC,(U), whence s centralizes T (cf. 
[9], K.I, 4.4). 


6. On p-extensions of nilpotent blocks 


6.1. Let G be a finite group and H a normal subgroup of G such that G/H is a p- 
group. In this section we will study the unique block of G lying over a nilpotent 
block of H (cf. [18], 1.7): the main tool to carry out our purpose are the semi- 
covering exomorphisms between G-algebras introduced in Sect. 3. Precisely, set 
A = 0Gand B = OH, and consider first A and B just as G-algebras: the key point is 
that the inclusion B c A determines a strict semicovering exomorphism, as we show 
below. Actually, this fact is useful to relate any block b of H with the unique block 
of G lying over b. 


Proposition 6.2. The G-algebra exomorphism determined by the inclusion B < A is a 
strict semicovering. In particular, for any pointed p-group P,. on A there is a non 
empty set S(y°) < P,(P) such that for any ye S(y°) we have y < y°, y is local if and 
only if y° is local, and then S(y°) = {¥*}.ec,(p)- Moreover, we have 


PA (P)= VU Sy’) (6.2.1) 
y €P,(P) 

Proof. By Theorem 3.16, to prove the first statement it suffices to show that, for any 
p-subgroup P of G, the 4-algebra exomorphism B(P)— A(P) induced by the 
inclusion B c A is a strict semicovering; but A(P) = 4C,(P) and B(P) = 4£C,(P) 
(cf. [19], 2.9.2), and the 4-algebra exomorphism above is determined by these 
isomorphisms and by the inclusion C,(P)< C,(P); so, it suffices to apply 
Example 3.9. Moreover, the second statement follows from 3.10, 3.3.2 and 3.15.3, 
and the equality 6.2.1 follows from 3.10 and the last statement of Proposition 3.3. 
Finally, the transitivity of C,(P) follows from 3.10.2 since A(P,:)°«) = 4 
(cf. [19], 2.9.2). 


The following consequence is more or less well-known. 


Corollary 6.3. We have P,(G) = P,(G). Moreover, let « be a point of G on A, P, a 
pointed p-group on B and y° the point of P on A such that y < y°; then P, is a defect 
pointed group of G, on B if and only if P,» is a defect pointed group of G, on A. 
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Proof. By Proposition 6.2, any idempotent of A has an A *-conjugate contained in 
B (cf. 3.10 and 3.2); in particular, as A® = Z(A), any idempotent of A®% belongs to 
BS and therefore P,(G) = PA,(G). Moreover, if P, is a defect pointed group of G, on 
B then « < Tr$(B”+y B”) and a fortiori « < Trp(A” + y° A”) which implies that P.,» 
is a defect pointed group of G, on A (cf. 3.15.3 and [15], Th. 1.2). Conversely, if P,,- is 
a defect pointed group of G, on A then P,. is a maximal local pointed group on A, 
and therefore P, is a maximal local pointed group on B (cf. 3.15.3) such that 
P, oc G, 


6.4. From now on, we consider A endowed with B as a G-algebra extension 
(cf. 2.9). Let « and B be respectively points of G and H on B such that H, c G,; by 
Corollary 6.3, « is also a point of G on A and the corresponding block of G lies over 
the block of H determined by £. Moreover, set N = Ng(H,) and let P, and Q; be 
respectively defect pointed groups of G, and H, such that Q; < P, < N, (cf. 5.2); by 
Proposition 5.3 we have 


Q=POH and N=P-H; (6.4.1) 


it follows from Corollary 6.3 that the interior G-algebra underlying the G-algebra 
extension A, (cf. 2.11) is an embedded algebra associated with G, on A 
(cf. [19], 2.13), and that the interior P-algebra underlying the P-algebra extension 
A, (cf. 2.11) is a source of the interior G-algebra underlying A, (cf. [15], Def. 3.2). 

The next result has already been proved — in an unessentially weaker form — 
in [3]. 


Proposition 6.5. With the notation above, « determines a nilpotent block of G if and 
only if B determines a nilpotent block of H. 


Proof. If « determines a nilpotent block of G and R, is a local pointed group on B 
such that R, < Hg, there is a unique e°¢ £P,(R) such that ¢ c e° (cf. Prop. 6.2) 
and we have RocG,; hence, Eg(R,o) is a p-group (cf. [18], 1.7.2) and 
Ny(R,) < Ng(R,), which implies that E,,(R,) is still a p-group. Conversely, assume 
that B determines a nilpotent block of H; if R, is a local pointed group on B such 
that R, < P, and if e°¢ YP,(R) contains « (cf. Prop. 6.2), it suffices to prove that 
E,(R,») is a p-group (cf. Prop. 6.2, Cor. 6.3 and [18], Th. 3.8). It follows from 6.4.1 
and 5.11.2 that the family {(P,, 1, u, P,)}, ¢p is a generator set of the N,-biexomor- 
phisms, and therefore it is easily checked that Ey(R,, P,) = Ep(R,, P,) (cf. 2.15.3); 
hence, as Ng(R,o)=Cg(R)*Ng(R,) (cf. Prop.6.2), we get Ng(R.»)= 
Cg(R): Np(R,) (recall that Eg(R,) = Ey(R,) by 5.2). 


The next result restates Theorem 1.12 in the present situation. Recall that, if S is 
a P-algebra, the interior P-algebra SP is the free S-module over P endowed with the 
evident multiplication and the canonical map from P (cf. 2.9). 


Theorem 6.6. With the notation above, assume that B determines a nilpotent block of 
H. There is an O-simple P-algebra S such that, considering SP endowed with SQ as a 
P-algebra extension, we have a unique P-algebra extension exoisomorphism 


A, = SP. (6.6.1) 
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Moreover, the P-algebra S is unique up to isomorphism and has a P-stable ©-basis 
which contains the unity as the unique P-fixed element. 


Proof. By Proposition 6.5, « determines a nilpotent block of G, and therefore as the 
interior P-algebra underlying A, is a source algebra of the block (cf. 6.4), it follows 
from the main theorem of [18] that there is an 0-simple P-algebra S, unique up to 
isomorphism, such that A, = SP as interior P-algebras and fulfilling all the 
conditions above. Moreover, we know (cf. [18], 1.8.1) that there is a unique group 
homomorphism p:P > S” lifting the action of P on S such that det(p(u)) = 1 for 
any ue P, and that SP = S ®, OP considering S endowed with p as an interior P- 
algebra; hence, since we have a (unique) embedding from the trivial interior P- 
algebra © to S @, S° (cf. [18], 5.7), we get an interior P-algebra embedding 


OP > S°@SP (6.6.1) 
o 

which is an embedded algebra (cf. 2.11 or [18], 2.10) associated with the unique 
local point of P on S° @, SP (cf. [18], 5.3.1). 


On the other hand, by Propositions 6.2 and 3.14 the inclusion B, < A, deter- 
mines a strict semicovering exomorphism of P-algebras. Then we claim that 


6.6.2. the inclusions S° @@ B, < S° @@ A, and § @gS° Ou B, < S @eS° Be A, de- 
termine strict semicovering exomorphisms of P-algebras. 


Since B is a direct summand of A as ©P-modules, for any subgroup R of P we get 
canonical inclusions 


(s° ® B,)(R)c (s° @A,)(R) and (S® s° ® B,)(R) < (S® s° @ A,)(R) 


and, by Theorem 3.16, it suffices to prove that these inclusions determine strict 
semicovering exomorphisms of 4-algebras; but the inclusion B,(R) < A,(R) does it 
(cf. Th. 3.16); hence, by Corollary 5.8 in [18], the following inclusions do it too 


S(R)° @ B,(R) < S(R)° @A4,(R) and 
& 


(S @ S°)(R) @ B,(R) < (S @ S°) (R) @ A,(R) 
0 & 0 & 


and now 6.6.2 follows from Proposition 5.6 in [18] (which is still true for P- 
algebras). 

Moreover, by 5.3.1 in [18] (which is also true for P-algebras) we know that P 
has unique local points S° x y on S° @, B, and S @,S° x y on S @gS° @oB, 
(where we still denote by y the unique point of P on B,). So, by 6.6.2, it follows from 
Proposition 3.3 that S° x y and S$ @,S° x y are respectively contained in the 
unique local points of P on S° @, A, and S @@S° @g A, (ef. 3.15.3 and [18], 5.3.1). 
In particular, since the interior P-algebra underlying the embedded algebra exten- 
sion (S° @g A,)so , yo S° @@ A, associated with Poo , ‘ (cf. 2.11) is an embedded 
algebra associated with the unique local point of P on S° @, A,, and since 
S° @¢ A, = S° @oS @e OP as interior P-algebras, it follows from 6.6.1 and the 
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uniqueness of embedded algebras (cf. 2.11.1 or [18], 2.10.1) that 
(S° ® A)so ,., = OP (6.6.3) 
© i 


as interior P-algebras. 

But such an isomorphism maps necessarily (S° ®@ B,)so , , onto Q; indeed, it 
is quite clear that P~™ B = Q and A = @,,, uB, where T is a left transversal to Q in 
P, which implies successively that 0: B, = B, and A = @,-7u° B,, that 

Q:(S°@B,)=S°@B, and S°@A,= Ou-(S°@B,), 
6 7 a C 


ueT 


and finally that 


Q-(S° ® B,)s0 . = (s° ® B,)so,, and (S° ® A,)s0 = ®u -(S° ® B,)s0 x. 93 
g ueT 

so, the image C of (S° @@ B,) so , , in OP through the bijective O-linear map 6.6.3 
fulfils Q°C = C and OP = @,,.7u°C which forces C = (Q. Hence, considering 
OP endowed with (Q as a P-algebra extension, the map 6.6.3 is an isomorphism of 
P-algebra extensions, and therefore we have 


S @(S° @ A,) so, , & SP (6.6.4) 
a C : 
as P-algebra extensions. On the other hand, the embeddings 0 > S @,S° (as 


interior P-algebras) and (S° @@ A,)so , —_ S° @, A, (as P-algebra extensions) in- 
duce the P-algebra extension embeddings 


S@(S°@ A,)sox, +S @S°@ A, <A, (6.6.5) 
ty O 


¢ G 
which are two embedded extension algebras associated with the unique local point 
S@ _S° x y of P on S @gS° @oB, (cf. 2.11). Consequently, by 6.6.4, 6.6.5 and 
2.11.1 we have A, = SP as P-algebra extensions. 

Now it suffices to prove that any P-algebra extension automorphism f of SP is 
induced by an element of ((SQ)”)*. Set 


W=1+4 ¥ Ker(tr)(1 — p(v)~*v) 
veQ 

where tr:S-— © denotes the trace map; by Proposition 6.8 of [18] we have 
W <(SQ)” and there is a unique we W such that f(S) = S”; thus P fixes w (since 
P normalizes W and f(S)). So we may assume that f(S) = S, and then there is 
séS* such that f(t) = t* for any te S; in particular, the uniqueness of p (cf. [18], 
1.8.1) forces fo p = p and therefore P fixes s. Hence, we may assume that f(t) = t 
for any teS, and then f = idgsp. 


7. The projective equivalence between nilpotent block extensions 


7.1. Let G be a finite group, H a normal subgroup of G and b a nilpotent block of 
H (cf. [18], 1.7). As we say in the introduction, our main purpose is to study the 
blocks of G lying over b. Clearly we have such a situation whenever H is a p-group 
and b = 1, and roughly speaking, our main results (cf. Th. 1.8 and Th. 1.12) state 
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that we can always reduce the general situation to a central p’-extension of such a 
particular case. In this section we give some characterizations of this reduction. But 
rather than to discuss just the reduction procedure, we will include it in a full 
equivalence relation which will allow us to invoke symmetry in all the arguments. 
This relation will be named projective equivalence to emphasize that central 
p’-extensions are neglected. 


7.2. Take again the notation introduced in 5.1 and 5.6. Similarly, let G’ be a finite 
group, H’ a normal subgroup of G’, A’ the G’-algebra extension formed by 
OG’ endowed with B’ = OH’, b’ a nilpotent block of H’, B’ the point {b’} of H’ on 
B’, «' the unique point of G’ on B’ such that H,. < Gj, N’ the normalizer of H. 
in G’, P’, and Qj. respective defect pointed groups of G, and Hj. 
fulfilling 0%, <P\,cN%, and N’ the quotient N’/H’. Recall that 
N’ = N,(Q}-)/Q’* Cs (Q' ) (cf 1.7.2) and that X’ denotes the image of X’ ¢ N’ in 
N’. As in 1.6, we consider respectively N and N’, endowed with the actions on the 
sets N and N’ induced by left multiplication, as N- and N’-permutation groups 
(cf. 2.4). 


7.3. Letag:N = N’ andt:P = P’ be group isomorphisms. We say that the G- 
and G’-algebra extensions A, and Aj, are projectively equivalent through o and t if 
the following conditions hold: 


7.3.1. We have G(u) = t(u) ) for any ueP. 


7.3.2. For any local pointed groups R, on B and R;, on B’ such that Q;< R, c P,, 
QQ, < Ri. < P', and t(R) = R’, the bijection from a P) to Hom . (R’, P’ ) 
induced by G ond t maps Ey x(R,, P,) onto Ey. 5(Ri-, P},). 


Then Aj, and A, are projectively equivalent oe o~* and t~'. We say that 
A, and A}, are projectivel 'y equivalent if there are group isomorphisms N = N’ and 
P = P’ fulfilling conditions 7.3.1 and 7.3.2. This relation is clearly symmetric and 
transitive, and does not depend on the choice of the defect groups. Moreover, 
condition 7.3.2 implies that: 





7.3.3. We have 6(Cy(R,)) = Cy (Ri). 

Indeed, by 7.3.2, for any ze€Cy(R,) there is x’eN’ such that (R;.)* < P},, 
t(u)* = t(u) for any ue P and x’~1a(n) = o(z~ 'n) for any ne N (cf. 2.5 and 2.8); 
therefore we have x’eCy(Ri-) (cf. Prop. 5.5) and x’=o(z), whence 
6(Z)eECy-(Ri.); now we get 7.3.3 by symmetry. 

7.4. First of all, we will describe the projective equivalence between A, and Aj, in 
terms of the full subcategories 2 and 2’ of the local categories of B, and B,. (cf. 2.8) 


formed respectively by the local pointed groups on B and B’ which contain defect 
pointed groups of H, and H;. Consider the evident functors 


f:2L>7MN and fF‘: 223 RN 


from 2 and @’ to the categories N and MN’ formed respectively by the N- and N’- 
permutation groups and the permutation group exomorphisms (cf. 2.5). Notice that 
any bijection from N onto N’ induces an equivalence of categories from M to N’. 
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Proposition 7.5. With the notation above, A, and Aj, are projectively equivalent if 
and only if there are a group isomorphism o: N = N’, an equivalence of categories 
t: 2 ~ 2’ and a natural isomorphism 7: 5° = f' ot, where s is the equivalence from 


MN to N’ induced by «a, such that 


7.5.1 for any object R, of 2, the N '~permutation group exomorphism 7(R,) has a 
representative inducing the identity on N’. 


Proof. Assume first that there are a, t and Tt as above fulfilling condition 7.5.1. Since 
t(P,) is a maximal object of 2’, we may assume that t(P,) = P},. We claim that, 
choosing a representative (t,, ids.) of t(P,) (cf. 2.5), the group isomorphisms 
o:N = N’ and t,: P = P’ fulfil conditions 7.3.1 and 7.3.2. By definition, for any 
ueP and any n’EN’ we have u-n’ = t,(u)*n’ (cf. 2.5), whence o(u)n’ = t,(u)n’ 
which proves 7.3.1; in particular, we get t,(Q) = Q’ (cf. Prop. 5.3). 


Moreover, let R, and Rj, be respectively local pointed groups on B and B’ such 
that Q;< R, c P,, Qc Ry. < P,, and t,(R) = R’, set t(R,) = Rz.. and choose a 
representative (t,,idy) = of t(R,). For any (N, N)-exomorphism (R,, x, P,)y, setting 


t((R,, x, Py Jw) =(Rox", Pye = amd —t((R,, 1, Pe) = (Revs y"s Py a» 
by the naturality of T we may assume that (cf. 2.5) 
t,(u)* =t,(u*) and t,(uy =t,(u) 
for any ue R, and that 
x"n’ = x°n' = 6(x)n’ and yn’ =n' 

for any n’eN’; hence we have (R”..)”"” = R’. (cf. Prop. 5.5) and, setting x’ = 
y”~*x", we get (Ri) < P!., G(x) = x’ and 1,(u*) = t,(u)* for any ue P, which 
proves that the bijection from Hom,(R, P) to Homg.(R’, P’) induced by & and %, 
maps (R,, x, P,)y upon (Rj, x’, P',.)x-. So, this bijection maps Ey 9(R,, P,) to 
Ey x (Ry, P;), and by symmetry its inverse maps Ey, 5-(R;-,P},) to 
Ey, w(Ry, Py). he 

Conversely, assume that there are group isomorphisms ¢o:N = N’, and 
t: P = P’ fulfilling conditions 7.3.1 and 7.3.2. For any object R, of 2 there are he H 
and xeN such that (Q;)" c R, and (R,)* < P,; notice that (Q;)* = Q; by Pro- 
positions 5.3. and 5.5. By condition 7.3.2 applied to Q; and Q%,,, the bijection from 
Hom,(Q;, P,) to Homy-(Q;,, P',,) induced by o and t maps (Q,;, hx, P,)y upon 
some element of Ey, 5-(Q5-, P’,,); that is, there is x’e N’ such that o(x) = x’ and 
t(v"*) = t(v)* for any ve Q. Moreover, by Proposition 5.5, there is a unique local 
pointed group R}. on B’ such that (R’)*’ = t(R*) and Q5, < (Ri.)* < P',, and we 
denote by t,: R > R’ the group isomorphism mapping ue R upon t(u*)* '; notice 
that by condition 7.3.1 we have 

t,(u) = t(u*)* = (u*)*' = o(u) 

for any ue R, and therefore (t,, ¢) is a permutation group isomorphism from (R, N) 
onto (R’, N’) (cf. 2.5). 

Now we consider the functor t: 2 > 2’ mapping R, upon Rj, as above and any 
morphism @€ Ey, x(R,, T,) upon the unique N’-permutation group exomorphism 
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t(@) from R’ to T’, where T,,, = t(7,), such that 
ale eae 
t(@)°(t,, G) = (ty, G)o Q , 

which belongs to Ey, 9-(R;, T,-) by condition 7.3.2; it is quite clear by symmetry 
that t is an equivalence of categories. Finally, denoting by s: 9 + M’ the equival- 
ence of categories induced by o, we have an evident natural isomorphism 
sof =f’ot mapping any object R, of 2 upon the N’-permutation group 
exomorphism determined by (t,, idy-). 


7.6. Now we will prove that the whole condition 7.3.2 depends only on the 
behavior under the action of ¢ and t of a “small set” of (N, N)-exomorphisms. 
Recall that for any xe N,(Q;) and any x’eN,-(Qj.) we denote respectively by 
P,(P,)* and P’,(P‘,,)* the unique local pointed groups (P 4 P*), on B and 
(P’ P’*’),. on B’ such that (cf. 5.10) 
(PaP"),< FP, and OP nk*}. <P. 

Notice that the family {P, .(P,)*, 1, x, P,)} where x runs over the set of elements 
of Ng(Q5) such that P,(P,)* =(P,a(P,/ (and so P,A(P,)* =(P,* OP,) 


and Po P* = Po P* is a generator set of the N,-biexomorphisms (cf. 2.16.1, 
2.17.1, 2.18.4 and 5.11.1). 


Theorem 7.7. With the notation above, let ¢:N = N’ and t:P =P’ be group 
isomorphisms and X a subset of Ng(Q5) containing Ng(P,) such that P, \(P,° = 

(P,A(P,"Y and PA P*= PaP* for any x€X. Assume that 6 and t fulfill 
condition 7.3.1 and that the family {( P, 0 (P,)*, 1, x, P,)}xex is a generator set of the 
N,-biexomorphisms. Then A, and A’, are projectively equivalent through o and t if 
and only if the following conditions hold: 





7.7.1. We have 6(Cy(Q5)) = Cy (Q5-)- 


7.7.2. For any x€X there is x'eNg(Q4-) such that P’\P’* =P’ AP’, 
P).n(P4. = (PY (PY), 6(X) = x’ and t(u*) = t(u)* for any ue P 2 P*. 


Proof. Conditions 7.7.1 and 7.7.2 are necessary; indeed, 7.3.3 implies 7.7.1; more- 
over, for any xeX, setting R,=P,A(P,)* and considering the (N, N)- 
exomorphism (R,, x, P,)z, condition 7.3.2 implies that there is x’eN’ such that 
a(x) =x’, t(u*) = t(u)* for any ueR and (Rj.)* cP’, where Rj, is the local 
pointed group on B’ such that Rj. c P',, and R’ = t(R) (cf. Prop. 5.5); in particular, 
we have 


(Ri) =Ri and  R’'=1(R) =6(R)=6(POP*)=P’nP* 
which implies 
PaP* =P \P* and = PLY oP oye. 
Conversely, assume that conditions 7.7.1 and 7.7.2 hold. Let R, and Rj. be 
respectively local pointed groups on B and B’ such that Q;- R,c P,, 
Q, < Ri. c P’,, and R’ = 1(R) (cf. Prop. 5.5), and y the unique point of R- Cy(R,) 
on B such that R, c R*Cy(R,),. We will prove first that: 





58 B. Kiilshammer and L. Puig 


7.7.3. If P, contains a defect pointed group of R*Cy(R,), we have 





o(Cy(R,)) < Cy (Re) - 


If 7, is a defect pointed group of R* Cy(R,),, it follows from 5.5.2 that T is a Sylow 
p-subgroup of R* Cy(R,), whence T 7 Cy(R,) is a Sylow p-subgroup of Cy(R,); but 
by 5.6.1 we get N7.,(R,) = T*Cy(R,) and therefore C,. ,(R,) = Cr(R)*Cy(R) 
(since C;(R,) = C7(R) by Proposition 5.5), whence TA Cy(R,) = C7(R). Hence, 
setting as in 5.6 (notice that Ker(7) = Cy(Q;)) 





N=N/C,(R,) and  N’=N'/Cy(R.) (7.7.4) 


and considering respectively Nand N’, endowed with the actions on the sets N and 
N’ induced by left multiplication, as N- and N’-permutation groups (cf. 5.6), it 
follows from Theorem 5.8 and conditions 7.3.1 and 7.7.1 that if 7, < P, we have 





&(Cy(R,)) = 6(Cp(R))*6(0"(Cx(R, N))) 
= t(C;(R))*O?(Cx(R’, N’)) c Cy (Ri) « 


Let (R,, y, P,) be an (N, N)-exomorphism; now we will prove that the N’- 


permutation group homomorphism (cf. 2.5) from (R’, N’) to (P’, N’) mapping t(u) 


upon t(u’) for any ueR and a(n) upon o(y~'n) for any ne N determines an 
element of Ey, 7-(Rj-, P',); that is, we will show that 


7.7.5. there is y'eN’ such that (Rj) < P',,,6(y) = y’ and t(u’) = t(u)” for any 
ueR., 


We may assume that P, contains a defect pointed group of R: Cy(R,),; indeed, 
anyway we have neN such that P, contains a defect pointed group of 
(R* Cy(R,),,)"; if we assume that 7.7.5 holds for any element of Ey s{(R,)", P,) and 
denote by Rj. the local pointed group on B’ such that Q4,c R?..c P}, and 
R” = 1(R"), then there are firstly n’eN’ such that (R%..)""'¢ P,,,a(n) =n’ and 
t(u) = t(u")""* for any ue R, whence R’.. = (R’.)" (cf. Prop. 5.5), and secondly 
y'€N’ such that (Ri) =(R%.)"'" oP.,ag(n- yy) =n’ "jy and t(u’) = 
t(u")""'”’ for any ueR. 

Consider the N,-biexomorphism (R,, 1, y, P,); by hypothesis, there are finite 
ordered families {x;};., of elements of X and {(R,,m;,P,(P,)“)}i-, and 
{(P,,;, P,)} ies of N-exomorphisms (cf. 2.8) such that (cf. 2.15.3) 


(R,, 1, Jy, F) = 2 mM;, PVP FY: 1, Xi» P,)(P,, Nn;, P,) $ 


iel 
Arguing by induction on |J|, we may assume that we have 
(R,, 3 y, P,) = (R,, 1, S, F.) + (R,, m, T,)(T,, 1, x, P,)(P,, n, P,) 


where xe X, T, = P,(P,)*,s = mn and y = mxn, and that there is s’'e N’ such 
that (R’,,)* < P’., (5) = s’ and t(u’) = t(u)® for any we R (cf. 7.7.5). Indeed, if i is 
the last element of J, we certainly may assume that y = m;x;,n; and, setting s = m;n,, 
it follows from the induction hypothesis that s = ztv where z€Cy(R,), ve P and 
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(R,, t, P,)v is an (N, N)-exomorphism fulfilling 7.7.5 (cf. 2.8); thus, there is t’e N’ 
such that (R‘.)" < P’., 6(f) =f’ and t(u') = t(u)" for any ue R; but by 7.7.3 we 
have o(z) = 2’ for some z’€ Cy.(R;,,); so, it suffices to set s’ = z’t’t(v). 

On the other hand, by condition 7.7.2 there is x’€N,(Qj-) such that 
Py A(PYy = (Py (PYF PO P’AP*® =P'OP* a(x) =x and t(u*) = t(uy 
for any ue T; notice that, setting T,,, = P’,, a(P’,,)*, we have 

t(T) = 6(P0 P*) = 6(P a P*) = P’ An P* =T' 
and therefore t(7) = 7” (cf. Prop. 5.3). Similarly, there is n’'e Ng.(P‘,,) such that 
a(n) =n’ and t(u") = t(u)” for any ueP. Set m’ =s'n’~' and y’ = m’x'n’; we 
claim that y’eN’ fulfils 7.7.5. Clearly o(m) = m’ and therefore a(y) = y’. More- 
over, as (R’,.)" < P', and Tc P’,, by Proposition 5.5 we have (Ry < Ty: 
(since R™ < T implies 1(R”) < t(T) = T’, and so R’™ =6(R™)=1(R")cT’), 
and therefore we get 
t(u”) = t(u™)" = t(u™y™ = t(u)” 
Let X’ be the set of x’'€ N,-(Q3,-) fulfilling the following conditions: 

7.7.6. We have P’,\(P‘,.)* =(P).a(P{,)* * and P’ 0 P’* 


7.7.7. There is x€X such that P,a(P, =(P,a(P,)* 
G(X) = x’ and t(u*) = t(u)* for any wet *(P' 9 P*), 


Now it suffices to prove that X’ contains N,(P‘,) and that the family 
{(P).0(P).)*,1,x', PS )} ex: is a generator set of the N%-biexomorphisms; 
indeed, in that case, we finish the proof invoking symmetry since by 7.7.7 the set X’ 
fulfils condition 7.7.2 with respect to ¢~' and t~'. By Theorem 5.8 and conditions 
7.3.1 and 7.7.1 we have (cf. 7.7.4) 


Ex(P,) = Ex, (P) = Eg:, g4P’) = Eq P,,-) 
and 








a(Cy(P,)) = o(Z(P))*a(O?(Cy(P,))) = t(Z(P))O?(CR(P',N’)) = Cy (P;,) 
Hence, for any n'e€N,.(P',,) there is ne Ng(P,) such that t(u)" = t(u") for any 
ueP, and by 7.7.2 there is z'€Cy-(P',,) such that a(n) = z’n’; since z’ = o(Z) 
for some z€Cy(P,), the element x = z~ ‘ne X fulfils condition 7.7.7 with respect 
to_n’ and therefore n’eX’. Moreover, by Corollary 5.11 the family 
{(P.a P*, 1, %, PVE) ux 8 generator set of the (N, N)- biexomorphisms, and 
therefore the family {(P’ 7 P’?™), 1, (x), P’)i-} vex iS a generator set of the 
(N’, N’ )-biexomorphisms; but it follows easily from 7.7.2 that o(X ) < X’; con- 
sequently, once again by Corollary 5.11, the family {(P‘,,0(P/.)*, 1, x’, P’ De ‘eX’ 
is a generator set of the N.-biexomorphisms. 


Remark 7.8. Actually, since we may assume that for any xeX thie N,-biexo- 
morphism (P,-(P,)*, 1, x, P,) is irreducible (cf. 2.16.1), a careful reading of our 
proof shows that in condition 7.7.2 we may replace o(x)= x’ by 
G(X)EG(Cy(P, 0(P,)) x’. 
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8. Source algebra extensions of extensions of nilpotent blocks 


8.1. In this section we will prove our main results, namely Theorems 1.8 and 1.12. 
Actually, in the proof of Theorem 1.12, we will give an explicit description of 
an 0-simple interior P-algebra S such that the P-algebra extension isomorphism 
1.12.1 holds. We take once again the notation introduced in 1.6 and 1.7. 


8.2. First of all we consider the case when G, H and 5b fulfil the following 
additional condition: 


8.2.1. There is an indecomposable OG-module M such that b-Res§(4 @@M) is a 
simple £Hb-module. 


Then A maps onto End,,(M) and, denoting by m the kernel of this map, we set 
S, = A,/f,*(m) for any pointed group K, on B (cf. 2.11) and denote byr,: A, > S, 
the canonical map. Clearly S, does not depend on the choice of f, in fi 
and we consider S,, endowed with the group homomorphism K — S ; determined 
by r,: A; > S; and the structural group homomorphism K - 4A ;, as an interior 
K-algebra, whence as a K-algebra extension (cf. 2.9). 


8.3. Let R, be a local pointed group on B such that Q; c R, c P,. As rs(Bz) = Sz 
(cf. 8.2.1), the homomorphism r,: B, > S, is surjective and we claim that it has a 
section; precisely, considering )',.r u*B, and S,R (cf. 2.9) respectively endowed 
with B, and S,Q as R-algebra extensions, we claim that 


8.3.1. There is an R-algebra extension isomorphism q,:S,R ~ Y yer U* B, such that 
r.(q,(s)) = s for any seé§,. 


Indeed, it is clear that }\,-p u* B, = (©(R*H)), as R-algebra extensions, and it 
follows from Theorem 6.6 applied to R:H, H and b that there is an 0-simple R- 
algebra S such that SR = (O(R:H)), (cf. Prop. 5.3 and Remark 5.4); in particular, 
r, induces a surjective R-algebra homomorphism from SQ onto S, and therefore an 
R-algebra isomorphism S = §, (cf. [18], 1.8.1). This proves also that: 


8.3.2. The p-group R stabilizes an ©-basis of S, containing the unity as the unique 
R-fixed element. 


8.4. Consider the R-algebra extension S° ®, A,; by Theorem 5.3 of [18], R has a 
unique local point S° x ¢ on S° @@B, and we denote by 


h,: C, + S°@A, (8.4.1) 
© 


an embedded algebra extension associated with S° x ¢, and by D, the structural 
subalgebra of C, (ie. D, = h; '(S° @ B,)); since we have an interior R-algebra 
embedding d,: 0 > S° ® S, (cf. [18], 5.7) and an R-algebra extension isomor- 
phism S,R = S, @ @ OR (cf. [18], 2.7.1), we have an R-algebra extension embedding 
OR > S$ @,S,R and it follows from 8.3.1 and the uniqueness of the local point 
S° x « that: 


8.4.2. There is an R-algebra isomorphism D, = OQ. 
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Another easy consequence of 8.3.1 is that 7,:B,+S, and _ therefore 
id @ 7,: S2 @~ B, + S° @~S, are covering exomorphisms of R-algebras (cf. [18], 
4.14 which can be extended to G-algebras) and, in particular, the unique local point 
of R on S° @@ B, maps to the unique local point of R on S° @,S,; hence, it follows 
from 2.11.2 applied to the composition of the R-algebra extension exomorphisms h, 
and id @7,:S° @, A, > S°@,S, that: 


_~_ ~ 


8.4.3. There i is an R-algebra extension homomorphism t,: C, — © such that d,ot, = 


(id @7,)oh 


85. As R,cP, there is a unique R-algebra extension embedding 
f: A,7> Res?(A, } suck that Resh( (f,) of? =f. (cf. 2.11.3); moreover, since 


fo (m) = (7) (f*(m)), 


the embedding f, ” induces an interior R-algebra embedding g?: S, > Res R(S,) such 
that g? 7, = Resk(7,) of’. ”. Consider the tensor product of these embeddings 


§? @f,’: S2@ A, + Resk(S°@ A,) ; (8.5.1) 
© © 


it follows from Proposition 5.5 that R has a unique local point on B, and therefore 
on S? @~B, (cf. [18], Th. 5.3); hence, the composition 


(G2 @f,’)oh,: C, + Resk(S2 @ A,) 
© 


is an embedded algebra extension associated with the second point; but by 8.4.2 we 
have D, = Resk(D,) and, in particular, ResR(h,); Resp(C,) > Resk(S° @@ A,) is 
an embedded algebra extension associated with the same point; therefore, by 2.11.1 
we get: 


8.5.2. There is a unique R-algebra extension exoisomorphism h?: C, = Resk(C y) 
such that (g? @f.’)> oh, = Resk(h, )oh?. In particular t,ch? =t,. 


The last equality follows from Resk(d,) = (g? @f,’) ed, (cf. [18], 5.7); indeed, we 
have (cf. 8.4.3 and 8.5.2) 
Resa(d,°t,) oh} = Resk((id @?,)°h,)> wis Resx(id @ F,)°(G2 @f)eh, 
= (G2 ® §2)°(id @?,) oh, = Resk(d,) °F, 
and we apply 1.3 and 1.5 of [17]. 
8.6. From now on we set C, = C, D, = D, S, = S, d, = d and t, = t, and choose 
C, = Resk(C) endowed with the embedding (h?)~! oh, denoted again by h, 


(cf. 2.11 and 8.5.2); moreover, we denote by t:P-—C™% the structural group 
homomorphism and it follows easily from 8.4 that 


D=¥ Or(v) and  t(P)C Nex(t(Q)) (1 + Ker(t)). (8.6.1) 
veQ 


In particular - N-«(t(Q)) normalizes D2=Z(D) and we denote by 
p: Ncx(t(Q)) > Nex(t(Q))/(D2)* the canonical map. Let x be an element of 
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N,(R,) and denote by g the automorphism of R induced by x (i.e. g(u) = u* for 
any ueR), and by 

f,: Res$(A) = Res, Res&(A) (8.6.2) 
the R-algebra extension isomorphism mapping a)¢€A on aj; as x normalizes R, 


and m, and R has a unique local point on S° @, B, (cf. 8.4), it follows from 2.11.1 
applied twice that: 


8.6.3. There are unique R-algebra extension exoisomorphisms 
fii: A, = Res,(A,), g&:S, = Res,(S,) and ht: Resk(C) = Res,Resk(C) 
such that 

Res,( ffi =feohs FoF, = Resy(F,)ofe and Resy(h,)oht = (G. @fe) oh, . 
But clearly Res*( f,) = id as -algebra exomorphisms, and therefore we get succes- 
sively Res®(f) = id, Res®(g‘%) = id (a priori true since S, is @-simple) and 
Res®(h®) = id as @-algebra exomorphisms (cf. [17], 1.5); that is, the maps f’, g% 


and h‘ are just conjugation by suitable elements of A,*, S;/ and C”, and the 
following proposition exhibits a particular choice of them. 


Proposition 8.7. With the notation and hypothesis above: 
8.7.1. There is ae A, such that f,(a)exB* and f£(a 9} = a4 for any ag€A,. 


8.7.2. There is ceC™* such that h‘(c)e(r,(a)~' @ a)(S° @@B,)2 and ni (cg) = ch 
for any co€EC. 


Remark 8.8. In particular aB, = f,~'(xB) and cD = h; '(S° @ aB,). Moreover, 
notice that t(u)° = h§(t(u)) = t(u*) for any ueR. 

Proof. Set f.(1) =i; since i and i* belong to « there is b’e(B*)* such that 
i** =i, and therefore there is a unique ae A," such that f(a) = xb’i (since 
i(xb’)~'xb’i = i); moreover, denoting by f,: A, > A, the inner @-algebra auto- 
morphism induced by a (i.e. f,(a9) = a6 for any ag€A,) we have 


Fe fal Qo) = (xb!) * faq) xb! = f.(f(ao))” 


(since if,(49)i = f,(do)), which proves that f, is an R-algebra extension homo- 
morphism and that f, = fY (cf. 8.6.3). 


Similarly, set h,(1) = j; since j and (g% @f)(j) belong to S° x « (the unique 
local point of R on S° @,B,) and we may assume that (g% @f2)(j) = jr ' 94, 
there is d’e((S° @,B,)*®)* such that j = j"(¢)''®4)’ and therefore there is a 
unique ceC~ such that h,(c) = (r,(a)~' @ a)d'j; as above, the inner (-algebra 
automorphism induced by c is an R-algebra extension homomorphism belonging 
to hé. 


Corollary 8.9. With the notation and hypothesis above, assume that R, = Q;. There 
is a unique group homomorphism 


w: Ng(Qs) > Nex(t(Q))/(D®)* (8.9.1) 
mapping x€ N,(Q;) upon p(c) and we have Ker(@) = Cy(Q). 
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Proof. By 8.7.2 we have t(v)° = h2(t(v)) = t(v*) for any veQ and therefore 
ce Ne~«(t(Q)). Moreover, if x’€Ng(Q;5), a’€ A; and c’eC” is another system 
fulfilling the conditions above (ie. f,(a’)ex’ B®, f2(a9) =a for any ayeA,, 
h;(c')e(rs(a’)”' @ a')(S3 @@ B,;)2 and h2.(cg) = cs for any cy€C), then we have 
f,(aa’')exB2x'’B2=xx'B2 and = (f2of,?)(ay) = aa” 


for any a)€A;, and similarly 


hs(cc')e(rs(aa’)”*@aa’)(S$@Bs)® and — (hE o hE) (co) = cy 
© 


for any cy€C. But it follows easily from 8.6.3 that f2 of? = f%. and h®.oh® = h*,., 
and therefore we may assume that f,2..(a9) = a@” and h?,.(co) = cf , which proves 
that p(c) does not depend on the choices of a and c (since x’ = x~! forces cc’ € D®), 
and that the map assigning to x the element p(c)e Nc *(t(Q))/(D2)” is a group 
homomorphism. 

Moreover, if x € C,(Q) we get necessarily ae BY and ce D® (cf. 8.7.1. and 8.7.2). 
Conversely, if x¢C,(Q) either x¢H or x¢C,(Q); in the first case, we have 
BoaxB= @ and we get successively B; 1aB; = @ and DacD = @ (cf. Remark 
8.8); in the second case, since h®(t(v)) = t(v*) for any ve Q, we have c¢ C2 (cf. 8.6.1); 
in both cases p(c) + 1. 


8.10. Come back to the general situation above (cf. 8.6). Since Ng(R,) < Ng(Q5) 
(cf. 5.5.1), we may consider (cf. 8.6.3) 


fe: As = Resy(Ay), 92: Ss = Resy(Ss) and hi’: Res5(C) = Res, Res5(C) 


where wy denotes the automorphism of Q induced by x, and it follows easily from 
2.11.1 and 2.11.3 applied twice that 


Resk(f2) off =frof®, Res(9i)°g5 = 95°92 and Resk(hi)=h2 (8.10.1) 


where g5: S; > Res§(S,) is the embedding induced by fi (cf. 8.2). Now we are able to 
compare (x) with p(c) (where a and c have been chosen from R,!). 


Corollary 8.11. With the notation and hypothesis above, we have w(x) = p(c). In 
particular w(u) = p(t(u)) for any ue P. 


~ 


Proof. By our choices of C, and C; (cf. 8.6), and by 2.11.1 we have Res§(h,) = 
(g; ® f)°h, and therefore it follows from Remark 8.8 that 


cD = h;'(S2 @f, '(xB)) = hs '(S2 @ fs ‘(xB)) 


(notice that we may assuine f, = f, of; and h, = (g5 @ f$)°h;). On the other hand, by 
the definition of m and Proposition 8.7, there is c’eC™* such that p(c’) = w(x), 
c'D = hs ‘(S3 @ fs ‘(xB)) (cf. Remark 8.8) and hi(co) = ch for any cg¢C. Hence 
cD =c'D and, since Res&(hi,) = =h® (cf. 8.10.1), there is de(D2)* such that 
c'dc~'e€Z(C); therefore c~'c'd belongs to (D2)* and so p(c) = p(c’) = w(x). In 
particular, if x ¢ R we may choose firstly a equal to the structural image of x in A,“ 
and secondly c = t(x), which proves the last equality (setting R, = P,,). 
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8.12. Set F = Im(o), 1 = p-'(F)a(1 + Ker(t)), J = 1 a(D2)* and Z = Z(Q). By 
Corollary 8.9 we have F = N,(Q;5)/Cy(Q); therefore, on one hand w(P) is a Sylow 
p-subgroup of F (cf. Prop. 5.3) and, on the other hand, by 1.7.2 we have a group 
homomorphism E 

nm: I +N (8.12.1) 


mapping cel upon x for some x € Ng(Q5) such that p(c) = w(x) (i.e. @ '(p(c)) = 
{x(c)}). By 8.6.1 and Corollary 8.11 we have 


u(P)cl and Jatr(P)=t(Z). (8.12.2) 


Moreover, notice that J is a p’-divisible abelian group (cf. 8.6.1). The following 
statement is the key step in our proof of Theorems 1.8 and 1.12. 


Theorem 8.13. Wéith the notation and hypothesis above, there is a finite subgroup L 
of I such that: 


8.13.1. We have t(P)< L,I =J-Land t(Z)=JOL. 


8.13.2. Considering the ©-algebra OL endowed with the group homomorphism 
t: P-(@L)* and the subalgebra ©(t(Q)) as a P-algebra extension, the inclusion 
L < C% induces a P-algebra extension isomorphism OL = C. 


8.13.3. The group L and the group homomorphisms x: L—> N and t: P > L fulfil 
conditions 1.8.1 and 1.8.2. 


Proof. Setting I = I/t(Z) and J = J/t(Z), and denoting by p: 1 F the group 
homomorphism determined by p, we have the following exact sequence 


bed ihc ei (8.13.4) 


and, since w induces a group isomorphism P/Z = w(P), t induces a group homo- 
morphism 7: w(P)— J such that p(z(w(u))) = w(u) for any ue P (cf. Cor. 8.11); 
that is, t fulfils condition 4.7.1 and we will prove that t fulfils also condition 4.7.2 
with respect to the elements fe F such that w(P) A (PY = (@(P) A w@(PYY. Let f 
be such an element and R the subgroup of P containing Q such that w(R) = 
w(P) (PY (notice that w(Q) is normal in F); by Proposition 5.5, there is a 
unique ¢€ #P,(R) such that Q; c R, ¢ P, and, if J is the unique point of R* Cy(Q) 
on B such that 0; R°C,(Q), (cf. 2.16.2), R, is a defect pointed group of 
R-Cy(Q), (cf. 5.5.2). Hence, by the Frattini argument, there is x €e Ng(R,) such that 
w(x) =f, and by Corollary 8.11 there is ce Nc«(t(Q)) such that p(c) = w(x) =f 
and t(u*) = t(u)° for any ue R (cf. Remark 8.8); as we may assume that ce/ (up to 
replacing c by t(c)~ ‘c), we get p(c) = f and t(w(u))’ = t(w(u)) for any ue R, where 
é is the image of c in J. 


It follows now from 2.17.1 and Corollary 4.7 that there is a group homo- 
morphism ¢: F >I such that pod = id, and o(@(u)) = t(@(u)) for any ue P. We 
claim that the inverse image L of Im(¢) in IJ fulfils conditions 8.13.1 to 8.13.3 above. 
Condition 8.13.1 follows easily from the properties of ¢. Moreover, we have 
n(L) = N = Ng(Q5)/O*Cy(Q) (cf. 8.12.1, 8.13.1 and 1.7.2) and, if {xz}ce, and 
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{yzhwen are respectively sets of ‘representatives for N in N,(Q;) and in L, it is clear 
that w(x;) = p(yz) for any ne N (cf. 8.12.1), and that Ag = ©; 9 XqB, (cf. 1.7); then 
it follows from Remark 8.8 that 


YaD = hs *(S; @fs *(xnBz)) 


for any ne N and therefore we get C = @z.7y,D, which proves condition 8.13.2. 
On the other hand L -q Ker(z) = 1(Q) (since LA J = t(Z)), Ker(t) = {1} (cf. 8.6.1 
and Cor. 8.11) and t(P) is a Sylow p-subgroup of L; in particular, in 7.2 we may 
assume that G’ = L, H’ = t(Q) and b’ = 1, and in 7.3 we may consider the group 
isomorphisms N = L/t(Q) and P = t(P) induced respectively by x and 1; then it is 
quite clear that conditions 7.3.1 and 7.3.2 imply conditions 1.8.1 and 1.8.2. So, it 
suffices to prove that A, and OL are projectively equivalent through the group 
isomorphisms N = L/t(Q) and P = t(P) above. 

Since z(t(u)) = u for any ue P (cf. 8.12.1 and Cor. 8.11) and 2(C,(t(Q))) = 
Cy(Q;5) (cf. 8.12.1 and Remark 8.8), conditions 7.3.1 and 7.7.1 hold and, by 7.6 and 
Theorem 7.7, it suffices to prove that condition 7.7.2 holds with respect to the 
elements x € Ng(Q;) such that P 4 P* = Pc P* and PL, O(P,Y = (P, 0 (P,). Let x 
be such an element and set R, = P, 0 (P,)*; by Corollary 8.11 and Remark 8.8 there 
is ce Nc«(t(Q)) such that w(x) = p(c) and t(u*) = t(u)’ for any ue R. Hence, since 
we may assume that ce/ (up to replacing c by t(c)~ 'c) and we have I = JL (cf. 
8.13.1), we get c = zy for some ye L and some ze€ J; moreover, as c normalizes t(R) 
and we have J 0 L = t(Z) c t(R), y and therefore z still normalize t(R) and, since 
N,(t(R)) = C,(t(R))*t(N2z(R)) by Lemma 8.15 below, we may assume 
that z centralizes t(R), whence p(z) = 1. In that case, we get n(y) = x (since 
p(y) = p(c) = w(x)) and t(u*) = t(u) for any ue R, and then it is quite clear that 
R = Po P* implies both t(R) = t(P) 0 t(P)’ and that the image of t(R) in L/x(Q) 
is the intersection of the images of t(P) and t(P/. 


Remark 8.14. The existence of ¢: FI above follows also from a result of 
Oliver [11]. Indeed, we may assume that 4 is an algebraic closure of the prime 
field and that © is totally unramified. In that case Oliver’s result implies that 
H'(w(P), J) = {0}. On the other hand, the existence of t shows already that the 
exact sequence 8.13.4 splits, and the existence of ¢ is equivalent to the existence of a 
complement L of J in I containing t(w(P)). But a complement of J in I contains 
certainly a complement of J in J-z(w(P)) and Oliver’s result forces the last one to 
be 7(w(P))? for some ze J. 


The following lemma, quoted above, is a slight generalization of a result by 
Coleman [5] and our proof employs his argument (see [17] for a different 
generalization). 


Lemma 8.15. Let K be a group, Q a subgroup of K and R a finite p-subgroup of 
Nx(Q). We have Nieg)«(R) = Ceg):(R)* No(R). In particular, if Q@<R then 
Nzog)*(R) am Czo9)*(R)° NzQ)(R). 


Proof. For any a€ N@g)(R) we consider the action of R on OQ mapping ce CQ 
upon u~ ‘ca~ ‘ua for any ue R; clearly R stabilizes Q (since Q = K 7 0Q in @K) and 
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fixes a through this action; so if a = » ial A,x where j,€0 for any x €Q, there is 
x €Q fixed by R such that A,.¢ 0* since the augmentation map 0Q — 0 assigns to a 
an invertible element; that is, we have a~'ua = x~ ‘ux for any ue R and therefore 
a = zx for some Z€ Cog)«(R). 


8.16. Henceforth we prove Theorems 1.8 and 1.12. In order to prove the first one 
and the existence of S in the second, we may assume that 0 is totally unramified (cf. 
[18], 2.13.1) and, in particular, that © contains a non-trivial root of unity € of 
p-power order only if p = 2 and € = — 1. Let n; be the unique ideal of B; such that 
B;/n; is an O-simple algebra and the determinant of the structural image of any 
veQ is one (cf. [18], Th. 1.6 and 1.8.1), and n, the unique ideal of B, such that 
(ff)~ * (1g) = nts (cf. 2.14.1). We claim that: 


8.16.1. For any xeEN, we have (ng)* = ng. 


Indeed, by 1.7.2 we may assume that xe N,(Q;) and, as in 8.6, we denote by 
f8: Res§(Ag) = Res,Resj(A,) the @-algebra extension isomorphism mapping 
ae A, upon a*, where ¢ is the automorphism of Q induced by x; then, by 2.11.1, 
there is a Q-algebra extension exoisomorphism f?: A; = Res,(A;) such that 
Res, (ff) o f° = fb ff; now the uniqueness of n, forces f?(n;) = 15, whence we get 
f2 (ng) = ng. In particular, it follows easily from 8.16.1 that G stabilizes the 
isomorphism class of an OHTr&(b)-module M such that 4 @,b-M is a simple 
4Hb-module. 


8.17. Thus there is always an extension G’ of G by an abelian finite group U’ such 
that H lifts to a normal subgroup of G’ — still denoted by H — and such that the 
triple G’, H and b fulfils condition 8.2.1. Moreover, denoting respectively by N’ and 
V’ the stabilizers of b in G’ and in U’, we may assume that 


V'c D(N')AZ(N’) (8.17.1) 


and that V’ is isomorphic to a subgroup of @ *. As in 1.6, set A’ = OG’, consider A’ 
endowed with B as a G’-algebra extension and denote by N’ the quotient N’/H. 
Clearly « is still the unique point of G’ on B such that H, c G;,, f is still a point of 
N’ on Band, denoting by P’ the Sylow p-subgroup of the inverse image of P in G’, y 
is still a local point of P’ on B such that Q; c P;, c Nz < G, and P, is also a defect 
pointed group of Gj. As in 1.7, denote respectively by 


fi: A > A, fj: Ag > Res$(A) ands’: Ay > Res$(A) 
embedded algebra extensions associated with G,, N, and any local pointed group 
R; on B such that Q; c R; c P,. Notice that, since A, > ON’b (cf. 1.7), Ag has an 
evident @V’-algebra structure and then A; has a unique OV’-algebra structure 
such that the canonical embedding f/: A; > Res}:(Aj) (cf. 2.11.3) becomes an 
embedding of R’-algebra extensions over OV’; thus, since A, = ONb (cf. 1.7), we 


have N- and therefore R-algebra extension isomorphisms, R being the image of R’ 
in P and © the trivial OV’-algebra, 


Ap=O@A, and A,=O@A; (8.17.2) 
OV’ oV’ 
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where © @ey Ag and © @ey A; are respectively endowed with the subalgebras 
1 @ B, and 1 @ B,, and with the evident group homomorphisms. 


8.18. Since G’, H and b fulfil condition 8.2.1, there is an ideal m’ of A’ such that 
A'/m’ is an 0-simple algebra and we have 


bA'b = Bb + bm’b, 


and we may assume that the restriction U’ +(A’/m’)* of the structural homo- 
morphism is injective. As in 8.2, we set S; = A;/f;~*(m’) considered as an interior 
R’-algebra, whence as an R’-algebra extension (cf. 2.9), and by 8.3.2 R’ stabilizes 
an (-basis of S; containing the unity as the unique R’-fixed element. Notice that the 
OV’-algebra structure of A; induces an OV’-algebra structure on S;, namely an 0- 
algebra homomorphism 


r: OV'36 (8.18.1) 
which does not depend on the choice of R,. 


8.19. As in 8.4, we denote by h/: C/ > S/° ®, A; an embedded algebra extension 
associated with the unique local point of R’ on S;° @,B,, by D; the structural 
subalgebra of C; and by t;: C; > © the R’-algebra extension homomorphism 8.4.3. 
The ©V’-algebra structures of A; and S; induce an OV’-algebra structure on 
S,° @, A; (through the diagonal map V’ > V’ x V’); clearly there is a unique OV’- 
algebra structure on C; such that hi becomes an embedding of R’-algebra exten- 
sions over OV’ and it is easily checked that t; is an OV’-algebra homomorphism. As 
in 8.6, we set C) = C’, D} = D’, S; = S’, d, = d’ and t, = t’, and we denote by 7’: 
v’-P’ +C’™ the structural group homomorphism (both as P’-algebra extension 
and OV’-algebra) and by p’: Nc-«(t'(Q)) > Nc-«(t'(Q))(D’2)* the canonical map. 


8.20. Now Proposition 8.7 and Corollaries 8.9 and 8.11 apply to our situation, 
and we denote by 


o': Ng(Qs) + Ne-=(t'(Q))(D'?)* 


the group homomorphism 8.9.1; notice that: 


8.20.1. For any v' EV’ we have w(v’) = p’(t'(v’)). 


Moreover, as in 8.12 we set F’ = Im(o’), I’ = p’*(F’)a(1 + Ker(t’)) and J’ = 
I'o Ker(p’), and denote by 2’:I’ + N’ the group homomorphism 8.12.1; notice 
that (cf. 8.20.1 and Cor. 8.11): 


8.20.2. We have t'(V’) cI’, J’'nt'(V’) = {1} and n'(t'(v’)) = v' for any v'eV’. 
Moreover, the complement of t'(V’ P’) in t'(V’) is the unique complement of 
J'st'(V AP’) in J'-t'(V’). 

Hence, by Theorem 8.13, there is a finite subgroup L’ of I’ such that: 

8.20.3. We have t'(V’:-P’)c L,I’ =J'-L' and t'(Z) = J'0L’. 


8.20.4. Considering the ©-algebra OL’ endowed with. the group homomorphism 
t':V’: P’ +(OL’)” and the subalgebra O(t'(Q)), the inclusion L’' < C’* induces an 
isomorphism OL’ = C' both as P’-algebra extensions and as OV'-algebras. 
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8.20.5. The group L' and the group homomorphisms 1’: L' + N' and t':P’ > L’ fulfil 
conditions 1.8.1 and 1.8.2 with respect to G’, H and b. 


8.21. We claim that the group L = L’/t'(V’) and the group homomorphisms 
n:L > N and t: P > L induced respectively by 2’ and 1’ fulfil conditions 1.8.1 and 
1.8.2 with respect to G, H and b. Indeed, it is quite clear that condition 1.8.1 with 
respect to G’, H and b implies condition 1.8.1 with respect to G, H and b. Moreover, 
let R, be a local pointed group on B such that Q; < R, < P, and denote by R’ the 
inverse image of R in P’. Recall that we consider respectively N and N’ as N- 
and N’-permutation groups (cf. 1.6), and let us denote by Hom, w(R’, P’)y- the set of 
N’-permutation group exomorphisms (g’,f’ ) from R’ to P’ (cf. 2.5) such that 
og (V'AR')<V'aP’ and f' (n'V') =f'(v’)V’ for any n’eN’, which clearly con- 
tains Ey, _(t’'(Ri, P,) (cf. 2.8); then, we have an evident map 


Hom,.(R’, P’)y. + Hom,(R, P) (8.21.1) 


and it is clear that Ey, 5.(R,, P,) maps onto Ey (R,, P,). Similarly, we consider 
respectively L and L’ as N- and N ‘-permutation groups (cf. 1.8.1), we introduce the 
set of N’-permutation group exomorphisms Hom,. (t'(R’), t'(P’))y containing 
E,, x(t'(R’), t'(P’)) and we have a map 


Hom,.(t’(R’), t'(P’))y: > Homy.(1(R), t(P)) (8.21.2) 


sending E,, ¢(t'(R’),t'(P’)) onto E, x(t(R),t(P)). Since the bijections tz: 
Hom,(R, P) = Homy(t(R),t(P)) and tp :Homag(R’, P’)y, = Homy(t’(R’), 
t'(P’)),. induced respectively by t and t’ are compatible with the maps 8.21.1 and 
8.21.2, condition 1.8.2 with respect to G, H and 5 follows from condition 1.8.2 with 
respect to G’, H and b. To finish the proof of Theorem 1.8 it suffices to discuss the 
uniqueness of L, x and 1; this uniqueness follows easily from Proposition 4.9 since 
conditions 1.8.1 and 1.8.2 imply respectively conditions 4.9.1 and 4.9.2 with respect 
to any subset of L. 


8.22. We will prove now the existence part of Theorem 1.12. Since r’ maps 
injectively V’ into ©* and p divides neither rank,(S’) (cf. 8.18) nor |L’:t’(P’)| (cf. 
8.20.5 and Remark 1.9), the composition @’ of the structural homomorphism 
L' > Indt(p.,(S" )*, where S’" is the restriction of S’ to t’(P’), with the determinant 
map maps injectively t’(V’ 7 P’) into ©*; but, as we assume that ( is totally 
unramified, 6’(t'(V’ 7 P’)) has always a complement % in the group of the roots of 
unity of ©; consequently, L” = 6’~'(%) is a complement of t’'(V’ 7 P’) in L’ and in 
particular, setting t’(P”) = t'(P’) 0 L”, P” isa complement of V’ - P’ in P’, so that 
P” = P. We denote by t”: P > L” the group homomorphism induced by t’ and this 
isomorphism, and by S” the restriction of S’ to P. Then, by Remark 1.13, it suffices 
to prove that we have 


(8.22.1) 


as P-algebra extensions. 
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8.23. First of all, notice that we have the following P-algebra extension 
embeddings (cf. 8.4 and 8.19) 


id@hi:S@C+S'@S°@A, and d@id:A,+S' @S°@A, (8.23.1) 
O © € © Y 


which are embedded algebra extensions associated with the unique local point of P’ 
on S’ @@S'° @oB, (cf. 8.4.2 and [18], 5.3.1 extended to P-algebras). Moreover, the 
OV’-algebra structures of A’, S’ and C’ induce successively unique ©V’-algebra 
structures on S’ @@ S’° @@ A’, and S’ @@C’ in such a way that d' @ id and id @ hi, 
become (V’-algebra embeddings. Consequently, by 2.11.1 and 8.20.4 we have an 
isomorphism A’, =~ S’@® @ OL’ both as P’-algebra extensions and as OV’-algebras, 
and by 8.17.2 we get a P-algebra extension isomorphism, © being the trivial 
OV’'-algebra, 


A,=O0® (s @ oL' ) ‘ (8.23.2) 
ov’ o 

8.24. But denoting by V” the complement of V’ nq P’ in V’ and considering OL”, 
endowed with the group homomorphism P > (OL")* induced by t” and with the 
image of OQ, as a P-algebra extension, it is not difficult to check that we have 
P-algebra extension isomorphisms 


Ss’ ® (© ® ot") ~0®@ (s ® o1") ~O® (s ® ot’) (8.24.1) 
( ov" ov" o OV’ O 

mapping s” @ (1 @ y”) upon 1 @ (s” @ y”) where s” eS”, y”e L” and ©” denotes the 
ring 0 endowed with the OV”-algebra structure mapping any v” € V" upon r’(v”)~* 
(cf. 8.18.1). Moreover, as 


bA’b/bm’b = Bb(Bam')b and &£@(Bb(Bam)b) = B,/J(B,) (8.24.2) 
© 


(cf. 8.18), the structural homomorphism N’—(bA’b/bm’b)" induces a group 
homomorphism o’:N’—>N (cf. 1.11) mapping any 6”eV” upon r’(v”)E4” 
(cf. 8.18.1), and therefore a_ group homomorphism o’:L’>L such that 
fico’ = 6’on' where L = Res,(N) (cf. 2.4) and 7: L > N is the canonical map; now 
it can be checked that we have a P-algebra extension isomorphism 


0" @ OL"=0,L° (8.24.3) 
ov" 
mapping 1 ® y” upon o’(y”)~' for any y”€L”. The isomorphism 8.22.1 follows 
now from the isomorphisms 8.23.2, 8.24.1 and 8.24.3. 


8.25. It remains to prove the uniqueness part of Theorem 1.12, without any 
hypothesis on the ramification of 0. First of all, we claim that 


8.25.1. if S is an O-simple interior P-algebra such that A, = S @o 0,L° then P 
stabilizes an O-basis of Indip)(S‘) where S* is the restriction of S to t(P). 


Indeed, for any x, yeL, it suffices to prove that 1(R) = t(P)Qt( PY ot(P)” 
stabilizes an 0-basis of x~' ® S* @ y, or equivalently, that R stabilizes an 0-basis of 
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Res§ *(a)(S) where 6,, ,:R > P x P maps ue R upon (u* ',(u” ')~*) (we identify R 
with Oy, y(R)); but, if x and p lift respectively x and y to L, it is clear that 
Yucr OPt(u)k~! is a direct summand of 0,L° isomorphic to Ind F* (Ry) as 
©(R* x R”)-modules and therefore Ind" Bess - R)(S)) is a direct pai of 
A, as ©(R* x R”)-module; in particular, Res? 5, “(R)OS) is a direct summand of 
Res? * er (R)(Ay ), SO it is a permutation OR- module. 


8.26. Let S and T be 0-simple interior P-algebras fulfilling the conditions of 
Theorem 1.12. By Remark 1.14, we know already that S and T are isomorphic as 
P-algebras; hence, we have 4 ®@S = 4 ®@ Tas interior P-algebras (since the lifting 
of the action of P is unique) and, in particular, 


k ® Ind“ p,(S*) = 4 ® Ind“ »)( 7") (8.26.1) 


where S‘ and T* are ie iil the restrictions of S and T to t(P). But it follows 
from 8.25.1 and from the fact that p does not divide the dimension of both terms in 
8.26.1, that the P-algebras Res,(Ind},p,(S")) and Res,(Ind?,p)( 7")) fulfil conditions 
1.3.2 and 1.3.3 in [20] and therefore they are isomorphic (cf. [20], Cor. 3.7). 
Moreover, since the determinant maps the images of P in Ind4p(S")“ and 
Ind‘ »,(7*)* onto {1}, we get 


Res, (Ind p,(S*)) = Res,(Indé »,(7")) (8.26.2) 


as interior P-algebras. As the canonical embeddings S — Res,(Ind‘p,(S")) and 
T > Res,(Ind*p,( 7")) (cf. 2.13.1) are embedded interior P-algebras associated with 
the unique local point of P on both terms of 8.26.2 (cf. [18], Cor. 5.8), we get finally 
S = T as interior P-algebras. 
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1. Introduction 


Let A be a commutative ring, and let J be an ideal of A. The cohomological 
dimension of I in A, denoted by cd(A, I), is the smallest integer n such that the local 
cohomology modules H#(M) = 0 for all A-modules M, and for all q > n. (Actually 
if H}(A) = 0 for all q > n, then cd(A, I) < n, cf. [Hal, Sec. 1].) If no such n exists, we 
set cd(A, I) = 00. However, if A is Noetherian of finite Krull dimension d, then 
cd(A, I) < d; a well-known result due to Grothendieck [Grl]. Another important 
vanishing result is that if J is defined set-theoretically by t equations, then 
cd(A, 1) St; in particular cd(A,1) provides a lower bound for the number of 
elements needed to define J set-theoretically. 

The cohomological dimension has been studied by several authors; for example 
see Faltings [Fal], Grothendieck [Gri], Hartshorne [Hal], [Ha2], Hartshorne 
and Speiser [H-S], Ogus [O], Peskine and Szpiro [P-S], and [S]. In this paper we 
will extend results from all of these articles. The main thrust of this paper is to give 
bounds on cd(A, 1), especially when I has small height (i.e., Spec(A/I) has small 
codimension) compared to the dimension of A. 

Hartshorne [Hal] proved an important result, widely known as the 
Hartshorne—Lichtenbaum vanishing theorem. This theorem basically states that if 
A is a complete local Noetherian domain, and / is an ideal of A then dim(A/I) 2 1 
iff cd(A, I) < dim(A). This theorem plays a crucial role in our proofs. In the same 
paper, Hartshorne proves a vanishing theorem for the cohomological dimension of 
the complement of a subvariety of projective space, which was later generalized by 
Ogus [O, Cor. 2.11] in equicharacteristic 0, and by Peskine and Szpiro [P-S, III, 
5.5] in equicharacteristic p > 0. Combining their results gives: 


Theorem 1.1. Let (A, m) be a regular local ring of dimension d, and let I be an ideal of 
A. Assume that A is complete and has a separably closed residue field k which it 
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contains. Then the following are equivalent: 


(i) cd(A, 1) <d—2. 
(ii) dim(A/I) = 2, and Spec(A/I) — {m} is connected. 


Of course one can remove the assumption that A is complete with separably 
closed residue field if one assumes in (ii) that Spec(A/I) — {m} is “formally geomet- 
rically connected”, which in essence reduces the problem to (ii) of Theorem 1.1. 

Ogus proves this theorem in equicharacteristic 0 by using the theory of 
algebraic De Rham cohomology developed by Grothendieck and Hartshorne, 
while Peskine and Szpiro use the action of Frobenius in characteristic p > 0. 

We are able (Theorem 2.9) to give a new proof of this result, independent of 
characteristic. The proof of Theorem 2.9 is based upon an induction theorem (2.5) 
which generalizes a result of Faltings [Fal, Satz 1]. The need to have k separably 
closed in both Theorem A and in our later work necessitates some work, which 
discusses the splitting of primes in power series rings when the coefficient field is 
extended. Although we need this work in Sect. 2, we postpone the proofs until 
Sect. 4, since they are independent of the rest of this paper. Our main theorem in 
Sect. 4 is Theorem 4.2, which says that if a prime P in a formal power series ring 
with coefficients in a field k splits into t irreducible components in the power series 
ring over a field L containing k, then there is a finite separable extension K of k, 
contained in L, such that P already splits into t irreducible components in the 
power series ring with coefficients in K. This result generalizes some of the results of 
Chevalley’s classic paper on power series rings [Ch]. 

The chief use of our induction theorem, however, is not to give a simpler and 
unified proof of Theorem 1.1, but is rather to give improved bounds for cd(A, I) for 
arbitrary Noetherian rings A and ideals J of A. 

Faltings [Fal, Korollar 2] gave the most general vanishing result for ideals J in 
Noetherian rings A. Note that in general cd(A, I) = sup{cd(Ap, Ip): PeSpec(A)}, 
so there is no loss of generality in assuming A is local, which we will normally do 
throughout the paper. Let (A, m) be a local ring. In [Fal] there is a bound for 
cd(A, 1) which in part depends upon the embedding dimension of A. Since this 
theorem, as well as many of the results in this paper are somewhat technical in 
nature (especially in the case that the base ring A is non-regular), we will restrict 
ourselves in this introduction to the case where A is a regular local ring. 

Accordingly, let (A, m) be a regular local ring of dimension d, and let J be an 
ideal with bight(J) = b. (Recall the big height of I = max{ht(P): P is a minimal 
prime of J }.) Faltings shows that cd(A, 1) < 4 — [(d — 1)/b]. In general this is the 
best possible result without additional assumptions (see [Ly1]). We are able to 
improve this result in several directions. We show in (5.1) that: 


(0.1) If J is formally geometrically irreducible, cd(A, 1) < d — 1 — [(d — 2)/b], 
and this is best possible for all b, d, with 0 <b <d. 


and, 
(0.2) If A/I is normal, cd(A, 1) < d — [d/(b + 1)] — [(d — 1)((b + 1)]. 


These theorems are based on work in Sect. 3. We also give a bound on cd(A, /) if 
one assumes that there is an integer t such that I, is generated up to radical by 
t elements for all primes P with dim(A/P) = 3. For instance if one knows that I is 
locally generated up to radical by t equations on the punctured spectrum, then 
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these results are applicable. We refer the reader to Theorem 3.10 for a precise 
statement of this bound. 

In Sect. 5 we specialize our work to regular local rings, give some examples, and 
try to extend Theorem 1.1 to the case where cd(A, I) < dim(A) — 3. Let (A, m) be 
a regular local ring containing a field. We are able to show under the assumption 
that 2 bight(/) + 2 < dim(A), and provided no three minimal primes of J add to an 
m-primary ideal, then cd(A, I) < dim(A) — 3. (The best previously known result in 
this direction was that provided 3 bight(/) < dim(A), then cd(A, I) < dim(A) — 3.) 
This condition is not necessary; however we give necessary and sufficient condi- 
tions for cd(A, I) < dim(A) — 3 in the case where the number of minimal primes of 
I is at most 5 (under the condition that 2 bight(/) + 2 < dim(A)). This work 
generalizes in this case the theorem of Faltings, and other results in this paper. 

Finally in Sect. 6 we apply our theorems to give vanishing results for the relative 
singular homology groups with complex coefficients of subvarieties of projective 
space. The fact that there is a relationship between the vanishing of local cohomo- 
logy and the vanishing of relative singular homology was developed by 
Grothendieck and Hartshorne (see [Ha3]). For example, we show that if Y4 < P2 
is irreducible then H,(P", Y;C) = 0 for i < [(n — 1)/(n — d)]. If Y is in addition 
normal, then H,(P", Y;C) = 0 fori < [(n + In —d + 1)] + [nf(n—d+1)]-1 
(cf. Theorem 6.2). If Y is locally set-theoretically defined by n — d equations, except 
on a Zariski closed set of dimension at most 1, then H,(P", Y; C)=0 for 
i < 2d —n +1, a generalization of the well-known result of Barth [Ba]. A little 
more work shows that in all of the above cases 2;(P", Y) are finite in the same range 
as H,(P", Y;C) = 0. 


2. An induction theorem 


Let (A, m) be a local ring containing a field, and let I be an ideal of A. We denote the 
completion of A by A%, and by B we denote the ring obtained from A* by 
separably closing its residue field and completing. Specifically, set B = ((A* )")*, 
the completion of the strict Henselization of A*. We note that B is a faithfully flat 
extension of A. 

If M is any finitely generated A-module, set N = M @,B. Since B is faithfully 
flat over A, we have that Hj(M)@, B= Hi,(N), and H{(M) = 0 if and only if 
Hj4,(N) = 0. Thus, in deciding whether H4(M) is zero, it suffices to consider these 
modules when the ring A is complete with separably closed residue field. In fact, it 
is necessary to incorporate this case in any assumptions made concerning the ideal 
I; e.g., certain vanishing results on the local cohomology of J are true if J remains 
prime after passing to B, but it is not enough to assume / is prime in A to obtain the 
same results. 

Our first result is a sharpening of [Fal, Satz1]. It is rather technical, but 
supplies us with the necessary inductive tool to prove our main results. The main 
improvements over the result of [Fal] come from the use of the 
Hartshorne-Lichtenbaum local vanishing theorem [Hal, Thm. 3.1] (which we refer 
to from now on as ‘HLVT’). First, however, we need a couple of lemmas. 


Definition 2.1. Let A be a local ring. By mdim(A) we denote min{ dim(A/Q)|Q is 
a minimal prime of A}. If I is an ideal of A, we let c(1) = embdim(A) — mdim(A/IA). 


Lemma 2.2. Suppose that A is a universally catenary local ring containing a field, and 
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I is an ideal of A. Then mdim(A) = mdim(A%), and c(I) = c(IA% ). If B is defined as 
above, then c(I) = c(IB). 


Recall that a ring A is universally catenary if it is Noetherian and any A-algebra 
of finite type is catenary [Mat, p. 84]. Also if a local domain A is universally 
catenary, then A is equidimensional. [Ra] 


Proof. Since embdim(A) = embdim(A%), to prove that c(I) = (IA%), it clearly 
suffices to prove the first statement of the lemma. (Note that A universally catenary 
implies that A/I is also.) Choose any minimal prime ideal P of A. Since A/P is 
a universally catenary domain, if follows that if Q is any minimal prime over 
PA”, then dim(A/P) = dim(A*/PA%) = dim(A*/Q). As minimal primes of A* 
contract to minimal primes of A, it follows that md m(A* ) = mdim(A), as required. 
For the last statement it suffices to assume that A=k[[x,,...,x,]], 
B=k*°[[x,,...,X,]], and I is a prime ideal of A. Then c(J) = ht(I), and 
c(1B) = bight(JB) = ht(I) by [Ch, Prop. 9a]. O 


Lemma 2.3. Let A be a universally catenary local ring, and suppose that P is a prime 
ideal of A and I is an ideal of A. Then, 

i) dim(A/P) + mdim(A,) = mdim(A). 

ii) c(1) 2 c(Ip), if P contains I. 


Proof. Choose a prime ideal Q’, minimal in A and contained in P, such that 
mdim(A,) = dim(Ap/Q’ Ap). Since A is catenary, we obtain that dim(A/P) 
+ mdim(A,) = dim(A/Q’) 2 mdim(A) which proves (i). 

By [V, Thm. 2.2], embdim(A) — embdim(A,) 2 dim(A/P). Hence we obtain 
from i) that (J) = embdim(A) — mdim(A/I) 2 embdim(A,) + dim(A/P) — 


(dim(A/P) + mdim(Ap/Ip)) = c(Ip). O 


Lemma 2.4. Let A be a Noetherian local ring, and let I be an ideal of A. Suppose that 
B is a faithfully flat local extension of A with dim(B) = dim(A). If H4,,(Mp) = 0 
for an A-module M and for all PeSpec(A) with dim(A/P)>r, then 
Hig o(M @, Bo) = 0 for all Qe Spec(B) with dim(B/Q) > r. 


Proof. As dim(B) = dim(A), we have that dim(B/mB) = 0 [Mat, 21.A], where m is 
the maximal ideal of A. Let Q be a prime of B with dim(B/Q)>r, and set 
P=Q0 A. Since B/PB is faithfully flat over A/P, and B/mB is 0-dimensional, 
dim(A/P) = dim(B/PB) 2 dim (B/Q)>r. Therefore H4,,(Mp)=0, and hence 
Hig .(M @, Bo) a His dMp) @4,Bg =0. O 


Theorem 2.5. Let A be a Noetherian local ring containing a field, and let I be an ideal 
of A. Let B = ((A*)")*. Set c = c(IB). Let M be any finitely generated A-module, 
and let n > c be an integer. Assume that for all integers s, with 1 < s < c — 1, and for 
all gq 2n-— s, the following hold: 


i) Hi,(Mp) = 0 for all PeSpec(A), such that I < P, and dim(A/P) > s + 1. 
ii) Hig (M @, Bp) = 0, for all Pe Spec(B) such that IB ¢ P, dim(B/P) = s + 1, 
and P + Q is primary to the maximal ideal of B for some minimal prime ideal 
Q of IB. 
Then H4(M) = 0 for all q =n. 


Proof. We first reduce to the case where A = Band J = 1B. Assume we have shown 
the theorem in the case where A is already complete with separably closed residue 
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field, ic, for A=B. The hypothesis i) of the theorem implies that 
Hiz,(M @ 4 Bp) = 0 for all primes Pe Spec(B) with dim(B/P) > s + 1, by Lemma 
2.4. Hence i) and ii) hold in B, and thus H4,(M @,B) = 0 for all q 2 n. As B is 
faithfully flat over A, we obtain that H7(M) = 0 for all q = n, as required. Thus we 
may assume that A is complete with separably closed residue field k < A. 

Set e = embdim(A). Since A is a quotient of k[[T,,..., T,]], upon replacing 
A by this ring, and J by its preimage, we may assume that A is regular. In this case 
c is just the big height of J (bight(/)), namely the largest height of any minimal 
prime of J. By descending induction on q we may assume that Hj(M) = 0 for all 
q > n, and we only need to show that H7(M) = 0. We induct on the dimension of 
A/I, over regular local rings A, and ideals J. The starting point is when dim(A/I) 
= 0. In this case c = dim(A), and H7(M) = 0 for n > dim(A). We claim that this 
induction shows that the module H7(M) must have support only at m. If not, then 
localize at a prime P’ minimal in the support of H7(M); we must show that the 
assumptions i) and ii) of the theorem are still satisfied for the ring Ap and for the 
ring C = (((Ap-)*)")*. Observe that c(I) = c(IAp-) by (2.3 ii), and c(IAp-) = c(IC) 
by (2.2) (A is complete, hence universally catenary) Let Q be a prime ideal of C with 
dim(C/Q) = s + 1. Set Q’ = QO A. Then dim(C/Q) < dim(Ap-/Q’ Ap-) (as Ap + C 
is flat with closed fiber “ fence 0) < dim(A/Q’). Hence dim(A/Q’) > s + 1, so 
that i) shows that fu,(Mg')=0 for all q2n—s. It follows that 
Hic,(M @,4Cg) = Ofor q 2 n—s. By our choice of Q, it follows that condition ii) is 
satisfied. Let Op €Spec(Ap-). If dim(Ap:/Qp- )>s+ 1, then dim(A/Q) > s + 1, and 
Hi,,(MQ) = 0 for q 2 n — s. Hence i) is also satisfied, which by induction shows 
that H4 j4,.(M p-) = 0 for q 2 n, contradicting the choice of P’. This proves our claim. 

Let D = (A/I) @,A be the complete tensor product. The natural surjection of 
A onto A/I induces an (A/I)-linear surjection of D onto A/I whose kernel A is 
generated by the e elements 7,® 1 —1@T7;, where A=k[[T7,,...,7,]]. Set 
N =D@,M =(A/I)@,M 

Since ID = (A/I)@,1 ¢ ‘A, there is a spectral sequence for the composition of 
functors, E34 = H®(H4p(N)) => H%,"%(N). Because D is a flat A-algebra, we have 
that H4p>(N) = D@,H4(M). 

Set d = dim(A/I), and consider the term, 


E>" = H4(D @,Hi(M)) > ESN). 


Since d+n>d+c2d+(e—d)=e, and since A is generated by e elements, 
H4*"(N) = 0, so E4"(N) = 0. 

All incoming differentials to E4" are zero, since they come from E4~""*"~! for 
r>1,and H4,"(D@,H7*"~ ‘(M)) =0, as by assumption H f(M) = 0 for q > n. 

The outgoing differentials land in E4*""~"~", and we claim that 
H4,°"(D @,H"~"~(M)) =0 for r= 2. Indeed, if r>c, thn d+r>d+c2 
dim(A) = e, and since 4 is generated by e elements, H4;*’ (any module) = 
Suppose that r<c, and set s=r-—1. In general H7~*(M) is the direct 
limit of modules L,, where {L,} is the collection of finitely generated sub- 
modules of Hj *(M). Hence H4,*'(D @,H7‘(M)) = lim(H4'"(D @,L,)) = 
lim(H% 4*+r (A/I) ®, L,)), since direct limits commute with the tensor product and 
are exact [R} 

Let P,,...,P, be the minimal primes of the support of L, in A for a fixed a, 
and let Q,,.. 0, be the minimal primes of A/I. We claim that the minimal 
primes of (A/I) @, L, are Q;; = nilrad(Q, ®,A +(A/I) ®,P;) ¢ D. To prove this, 
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we first observe that by Corollary 4.5 the ideals Q;; are prime, since k is separably 
yes and is the residue field of A and A/I. Note that A and A/I embed into 

= (A/I )@,A. We have that Qj (D/P; D) = nilrad(Q,(D/P;D)) is a prime ideal, 
a its contraction to A/I is Q; since the map from ‘A/I to D/P;D = (A/I) @,(A/P; ) 
is faithfully flat. It follows that ‘the ideals nilrad(Q (D/P;D)) are exactly the minimal 
prime ideals of D/P, D, since any minimal prime of D/P,D must contract to one of the 
Q; (by flatness). As D is flat over A we see that Ass(D @ ,L,) = Upc ass i.) Ass(D/PD) 
by [Mat, 21.B]. Any minimal prime of D@,L, is an associated prime of this 
module, and therefore is associated to D/PD for some Pe Ass(L,). In particular, if 
N is a minimal prime of D@,L,, then N contains P;D for some i. By our 
calculation above, this implies that N contains 2;; for some i. However, all of the 
Q;, are associated to D @, L, by the formula above. Hence N must be equal to one 
of the Q;;, which proves our claim. 

We know that dim(A/Q,) < d = dim(A/I), and by i) dim(A/P;) Ss + 1 =r, 
and hence dim(D/Q;;) << d+ r. Suppose that dim(A/P;) <r for every i. Then 
dim((A/I) @, L,) < d +r, and so H4,*"((A/I) ®,L,) = 0, as we claimed. Thus we 
may assume that dim(A/P;) = r for some i. Then ii) guarantees that P; + Q; is not 
m-primary for every j. Hence in D, the ideal 4 + 2;; is not n-primary for any j, 
where yn is the maximal ideal of D. Thus 4 + Q;; is not n-primary for any minimal 
prime 2; of D having dimension equal to d+r= dim(C ) where C= 
(A/I) ®,(A/ann(L,)). Now apply HLVT [Hal, Thm. 3.1]; C is complete, so we 
may conclude that H4,*"(M)=0 for any module M. In particular, H4,*’ 
((A/I) ®, L,) = 0 in this case. Thus in any case the outgoing differentials are zero as 
well, and we may conclude that H4(D @ ,H'(M)) = 0. Since H7(M) is supported 
only at the maximal ideal, it is the direct limit of modules L having finite length. 
For any such L, we have that H4(D @,L) = H4((A/I)@,L) = H4((A/I) @,L). 
The support of (A/I) @,L can be identified with Spec(A/I). The intersection of 
A with this Support is the closed point {m/I}, where m/I is the maximal ideal of A/I. 
Hence, H4((A/I) @,L) = Hy (A/T) @,L) = H4,,(A/I)®@,L, where the latter 
isomorphism follows as L is flat over k. Passing to the direct limit, we obtain that 
H4(D @,Hi(M)) = H4,,(A/I) ®,Hi(M) =0. However, since H4,,(A/I) +0 
[Gr1, 6.4], and the tensor product is over a field, we conclude that H7(M) = 0, as 
required. O 


Remark 2.6. The hypotheses of Theorm 2.5 are fulfilled if for all s between 1 and 
c—1 and for all q2n—s, the module Hj, (Mp) =0 for all primes P with 
dim(A/P) = s + 1. This is Faltings’ original dhoveten [Fal, Satz 1]. 


Remark 2.7. Assume that nilrad(/B) is prime, in the context of Theorem 2.5. Then 
no prime P2>/B such that dim(B/P) =s + 1(1 <s <c-—1), and such that 
Hiz,(M @ , Bp) + 9, satisfies the condition that nilrad(/B) + P is primary for the 
maximal ideal of B. Hence condition ii) of Theorem 2.5 is vacuous in this case, and 
so the hypotheses of Theorem 2.5 will be satisfied if H7, (Mp) = 0 for all primes 
P with dim(A/P) = s + 2, and for all g = n — s. This ehervadinn will allow us to 
improve Faltings’ estimate in case I is formally geometrically irreducible (i.e., 
nilrad(/B) is prime). 


Remark 2.8. If c < 1 there are no integers s with 1 < s < c — 1. Hence (i) and (ii) 
are vacously satisfied. Hence Hj(M) = 0 for q > c. This can also be seen directly. 
Indeed if c = 0 then A is regular and J = 0. If c = 1 then A% is the surjective image 
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of a regular ring A’ of dimension e = embdim(A), and the preimage I’ of IA in A’ 
has bigheight 1. Hence es is principal and the result follows. 


To finish this section we wish to illustrate how Theorem 2.5 allows a new 
characteristic-free proof of theorems of Hartshorne and Ogus in characteristic 0, 
and Peskine and Szpiro in characteristic p [P-S]. These theorems give necessary 
and sufficient conditions for an ideal J in a complete regular local ring of dimension 
d with separably closed residue field to satisfy H4~1(M) = 0 for all modules M. 


Theorem 2.9. Let ( A, m) be a regular local ring of dimension d containing a field, and 
let I be an ideal of A. Then H4(M) = 0 for all q = d — 1 if and only if 


i) dim(A/I) = 2, and 
ii) Spec(A/I) — {m} is formally geometrically connected, i.e., If B = ((A*)")*, 
then Spec(B/IB) — {mB} is connected. 


Proof. By the remarks in the beginning of this section, we may assume that 
A=B=k[[x,,...,X,]], with k separably closed. First we will show the necessity 
of conditions i) and ii). If J is m-primary, then H4(A) + 0 [Grl, 6.14]. If I has 
a minimal prime over-ideal P of dimension 1, then H{, '(Ap) + 0, and therefore 
H4~'(A) + 0. Thus dim(A/I) = 2. Assume that Spec(A/I) — {m} is disconnected. 
Then there are two ideals J and K such that J q K has the same radical as J, both 
J and K contain J but do not have the same radical, and J + K is m-primary. By 
HLVT, H4(M) = H%(M) = 0 for any module M. The Mayer-Vietoris sequence of 
local cohomology, applied to J and K, shows that H4#~'(A) maps onto 
H4,x(A) + 0. Thus condition ii) is also necessary. 

For the sufficiency of conditions i) and ii), we first consider the case where I is 
prime. Set n = d — 1. By Theorem 2.5 and Remark 2.7, to see that H}#(M) = 0 for 
all q 2 n, it suffices to prove that H7, (Mp) = 0 for all q 2 n —s, and all P with 
dim(A/P) = s + 2, for 1 < s < bight(J) — 1 = ht(/) — 1, since I is prime. However, 
dim(A/P)=s+2 implies that dim(Ap)=d—s—2=n—s—1. Hence if 
q 2n-— s, then q > dim(Ap), so that H7, (Mp) = 0. 

Now consider the general case. Let P,,..., P, be the minimal primes over I, 
and use induction on r to prove the theorem, the case r = | being done above. 
Every P; must have dimension at least 2; if not, say P, has dimension 1. Then 
P, +()\i+:P; is m-primary, and this gives a disconnection of the punctured 
spectrum of A/I, in contradiction to assumption ii). Consider the graph TI with 
r vertices 1,... ,r, where i and j are joined by an edge if and only if P; + P; is not 
m-primary. Assumption ii) shows that I’ is connected. Any finite connected graph 
has a spanning tree [B, Cor. 5, p. 7]. The number of edges in a tree is smaller than 
the number of vertices, so there is a vertex from which only one edge emanates [B, 
Cor. 7, p. 8]. The original graph minus this vertex is connected. Hence we may 
assume that the graph I — {1} is connected, and set J = ();+;,P;. The ideal 
J satisfies the assumptions of the theorem, so by induction, H$(M) = 0 for all M, 
and for all gq =d-— 1. Also H},(M) =0 for all g 2d — 1. The Mayer-—Vietoris 
sequence applied to J and P, yields an exact sequence, H 4-1(M)@® 
H4-'(M) > H4~!(M) — H4,p,(M). Since J + P, is not m-primary, H4,p,(M) =0 
by HLVT. Hence H4~'(M)=0. By HLVT, H4#(M)=0, while Hj(M)=0 
forg2d+1lasdim(A)=d. O 
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3. Vanishing theorems 


In this section we prove our main vanishing results for local cohomology. They are 
fairly easy inductive consequences of our Theorem 2.5. Let (A,m) be a local 
noetherian ring containing a field, let J be an ideal of A with c = c(IA“”) > 0, and 
M be a finitely generated A-module of dimension d. G. Faltings [Fal, Korollar 2] 
proved that 

H}i(M)=0 = forqg>d-—([(d-—1)/c]. (3.1) 


Faltings also proved (cf. [Fal, Korollar 4]) that if Jp is generated set-theoret- 
ically by r elements for every prime P 2 / with dim(A/P) 2 2, then 


Hi(M)=0  forg>c+r-—1. (3.2) 


It is known that without further restrictions, (3.1) is best possible [Ly1]. In this 
section we sharpen these results by putting additional assumptions on the ideal J. If 
I is formally geometrically irreducible (see Definition 3.6 below), we are able to 
show (cf. Theorem 3.8) that provided 0 < c(I[A“”) < dim(A) then with d = dim(M), 


Hi(M)=0  forg>d—1—[(d—2)/c]. (3.3) 


In Sect. 5 we will show that (3.3) is best possible. In Theorem 3.9 we generalize 
the vanishing theorems further by assuming more about J. Let A be an excellent 
local ring containing a field, let J be formally geometrically irreducible with 
c(I[A*) = c, and assume that (A/J)p is normal for all primes P containing I with 
dim(A/P) = 3. If M is a finitely generated A-module of dimension d then, 


Hi(M) =0 for q >d — [(df(c + 1)] — [(d — 1)(e + 1]. (3.4) 


We are also able to generalize Faltings’ result (3.2). We show in Theorem 3.10 
below that if A is a local ring containing a field and / is a formally geometrically 
irreducible ideal such that J, is set-theoretically generated by r elements for all 
primes P containing J with dim(A/P) 2 3, then 


Hi(M)=0 for allg>c+r-—1, for all A-modules M . (3.5) 
We begin with a definition. 


Definition 3.6. We say that an ideal J in a local ring A is formally geometrically 
irreducible if B/IB has a unique minimal prime, where B = ((A“ )*")* ), the comple- 
tion of the strict Henselization of the completion of A. 


Our main technical tool of this section is (3.7) below which is a consequence of 
our induction theorem (2.5). 


Lemma 3.7. Let A be a local noetherian ring containing a field and let I be a formally 
geometrically irreducible ideal of A. Let n > c = c(IA“%) be an integer. Let M be any 
finitely generated A-module. Assume for all integers s, 1 < s < c — 1, and all primes 
P 21 with dim(A/P) = s + 2 that the local cohomology modules H4,,(M p) = 0 for 
all q 2 n-—s. Then H4(M) = 0 for all q 2 n. 


Proof. Set B = ((A”)")*). By assumption nilrad(JB) is prime. Hence there does 
not exist any prime P in Spec(B) such that P contains /B, dim(B/P) > 0,and P + Q 
is primary to the maximal ideal of B for some minimal prime Q of IB. This gives 
that condition (ii) of Theorem 2.5 is vacuously satisfied. Hence all of the hypotheses 
of (2.5) are satisfied and the conclusion of (2.5) gives the result. O 
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Theorem 3.8. Let A be a local ring containing a field, let I be a formally geometrically 
irreducible ideal in A with 0 < c(I[A”) = c < d, where d is the dimension of a finitely 
generated A-module M. Then H#(M) = 0 for q > d — 1 — [(d — 2)/c]. In particular 
if 0 < c < dim(A), then cd(A, I) < dim(A) — 1 — [(dim(A) — 2)/c]. 


Proof. Set B= ((A”*)")*. Since A” is universally catenary, (2.2) shows that 
c = c(JB), and the remarks at the beginning of Sect. 2 then show that we may 
assume that A = B. If c = 1, then ./JB is principal (see (2.8)), and the result is 
trivial. Henceforth we assume that c = 2. Set n(t) = t — [(t — 2)/c]. Then n is 
a nondecreasing function of t. In particular for t = d, using that d = c + 1, we see 
that n = n(d) > c. We apply (3.7) with this n. Choose any s, 1 < s < c — 1, and let 
P be a prime containing I with dim(A/P) = s + 2. A is complete, hence universally 
catenary, so (2.3) shows that c(Ip) < c. As dim(M,) < d — s — 2, (3.1) implies that 
Hi,(Mp) = 0 for g 2 d — s — 1 — [(d — s — 3)/c]. To apply (3.7) it suffices to show 
that n—s2d—s-—1-— [(d—s-— 3)/c], which reduces to 1 2 [(d — 2)/c] — 
[(d — s — 3)/c]. As s+ 1 Sc the inequality follows. We can now apply (3.7) to 
obtain the first conclusion of the theorem. The last statement follows from setting 
M =A and using [Hal, Sec. 1]. O 


Theorem 3.9. Let A be an excellent local ring containing a field. Let I be a formally 
geometrically irreducible ideal of A with 0 < c = c(IA%), and assume that (A/I)p is 
normal for all primes P >I with dim(A/P) = 3. Let M be a finitely generated 
A-module of dimension d. Then H#(M)=0 for q2d+1-—[df(c+1)]—- 
[(d — 1)(c + 1)]. In particular if n = dim(A) and 0 < c then 


cd(A, 1) Sn — [nc + 1I)] —[(n— Ic + 1]. 


Proof. We induct upon d to prove the theorem. If d < c + 1, the result follows from 
(3.1). Henceforth we assume that d > c + 1, and the result holds for modules of 
dimension < d. If c = 1, the conclusion holds (by (2.8)). Thus we also assume that 
c 22. We again apply (3.7). Set n=d+ 1 — [d/(c + 1)] —[(d@— Dc + 1)). To 
apply (3.7) we first show that n> c. This reduces to proving that d+1> 
(2d — 1)(c + 1) + c. Clearing the denominator yields the equivalent inequality 
c?—cd+d—2<0, ie. that (c — 1)(c + 1 — d) <1 which holds by our con- 
straints upon c. Thus n > c. 

Let 1 < s <c — 1, and let P > J bea prime with dim(A/P) = s + 2. By (2.2) and 
(2.3) c(I(Ap)* ) = c([p) S c. The dimension of Mp S d — s — 2. As dim(A/P) 2 3, 
A/P is normal. Since A is excellent (Ap/Ip)* is also normal and (4.4) shows that Ip is 
formally geometrically irreducible. We may apply the induction to Ap, Ip, and Mp 
to obtain that Hi,(Mp)=0 for q2(d—s—2)+1-—[(d—s—2 (c+ 1)] 
—[(d —s — 3)(c + 1)]. Now (3.7) will finish the proof if we show that 
n—s2=(d—s—2)+1-—[(d—s—2)(c + 1)] —((d—s — 3)(c + 1)]. This re- 
duces to 22 (([d/(c + 1)]—[(d—s— 2c + 1)}) + (Ld —- fe + 1) - 
[(d — s — 3)/(c + 1)])) which is true since s + 2 < c + 1. The last statement of the 
proof follows by setting M=A. O 


Finally we give our last vanishing result. 


Theorem 3.10. Let A be a local ring containing a field, let I be a formally geomet- 
rically irreducible ideal of A with c = c(1A“), and assume that Ip is set-theoretically 
generated by r elements for all primes P 2 I with dim(A/P) = 3. Then H}(M) = 0 for 
all q>c+r-—1, and for all A-modules M, i.e. cd(A, 1) Sc+r-—1. 
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Proof. Set n= r + c, and apply (3.7). For the last statement set M= A. () 


Remark 3.11. Let A be a universally catenary local ring containing a field, let J be 
an ideal of A with c = c(JA”), and let M be a finitely generated A-module of 
dimension d. Set c(v) = max{c(Ip): P 2 I, dim(A/P) 2 v}. By (2.3) the function c(v) 
is non-increasing. All of the results of this section were proven assuming only that 
c(0) is known. However, if more information of c(v) is available, stronger results are 
possible. For example: 

(i) Assume that there is a positive integer t < c = c(0) such that c(t + 1) St. 
Then H4(M) = 0 for g >d —1—-—[(d —1—-c)/t]. 

(ii) Assume that J is formally geometrically irreducible and there is a 
positive integer t<c=c(0) such that c(t+2)<t. Then H#(M)=0 for 
q>d—2-—[(d—c —2)/t]. 

(iii) Assume that A is excellent, J is formally geometrically irreducible, Ap/Ip is 
normal for every prime P 2 | with dim(A/P) = 3, and there exists a positive integer 
t <c such that c(t + 2) < t. Then H}(M) = 0 for q — 2 — [(d —c — 1)(t + 1)]- 
[(d —c — 2)/(t + 1)]. 

We omit the proofs since no new ideas are involved. Of course, if precise values 
of c(v) are known, even sharper estimates are possible. Also, if in addition to 
information about c(v) it is known that J, is set-theoretically generated by r ele- 
ments for primes P > | with dim(A/P) = m, where r and m are some fixed integers, 
even sharper results are possible, but the formulas are quite cumbersome. We 
conclude this section with several examples to illustrate the strengths of the various 
estimates. 


Example 3.12. Let k be a field of characteristic 0, and let X = (x;;) be a generic 
3 x 6 matrix. Consider the ideal J generated by the 3 x 3 minors of X in the ring 
B=k[x;;]. Let A = B,,). Then dim(A) = 18, w(/) = 20, and ht(I) = cI) = 5. 
Furthermore (e.g. see [B-V]), A/I is normal, Cohen—Macaulay, and if P + (x;;), 
L(Ip) S 10. We apply our various estimates to compute cd(A, 1). HLVT gives that 
cd(A, 1) < 19. Faltings estimate (3.1) gives that cd(A, I) < 18 — [17/5] = 15. Since 
I is homogeneous it is formally geometrically irreducible so that our Theorem 3.8 
gives cd(A, 1) < 18 — 1 — [16/5] = 14. Since A/I is normal and / is formally 
geometrically irreducible we may also apply Theorem 3.9 to conclude that 
cd(A, I) S 18 — [18/6] — [17/6] = 13. Finally we can apply (3.10) to obtain that 
cd(A, I) <5 + 10 — 1 = 14. Obviously (3.9) gives the best estimate. 

The actual cohomological dimension of J, however, is 10, by the remark after 
this example. This can be obtained by our methods, or even Faltings’ original 
theorem by using induction upon the size of the matrix. Clearly, though, the 
theorems above are not always the best possible—this is because information 
concerning localizations of an ideal will not in general tell if J is defined up to 
radical by a small number of equations. 


Remark 3.13. Let k be characteristic 0 and let J be generated by the r x r minors of 
an rxs generic matrix X (r<s). Let B=k[x,], and A=B,,). Then 
cd(A, I) = cd(B, 1) = r(s — r) + 1. 


Proof. We sketch the proof, which was probably known. If I is ideal in a poly- 
nomial ring generated by homogeneous equations of the same degree, then the 
analytic spread of J is the dimension of k[I] as a subring of the polynomial ring. In 
the case of this example, k[J] is just the homogeneous coordinate ring of the 
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Grassman variety of r-planes in s-space, which is known to have dimension 
r(s — r) + 1. Therefore, J is generated up to radical by r(s — r) + 1 equations, so 
that cd(A, I) S r(s —r) + 1. 

On the other hand, as Hochster pointed out to us, the map from C = k[J] to 
B = k[x;;] splits as a C-module since C is the ring of invariants of a linearly 
reductive group acting linearly on B. Hence the local cohomology H{,(C) 
injects into H#-(B) for all q. In particular, H7(B) cannot be zero for 
q = dim(C) = r(s —r) + 1, and consequently H4#(A)+0. Thus r(s—r)+1< 
cd(A, I), which proves the remark. O 


Suppose that we assume in this example that chark = p > 0. Since A/I is 
Cohen-Macaulay, [P-S] shows that cd(A, I) = ht(/) = 5. 


Example 3.14. To obtain a better idea of the various estimates, let us consider 
a complete regular local ring (A, m) of dimension 60 containing its separably closed 
residue field, and let I ¢ A be an ideal with bight(J) = 5. HLVT shows that 
cd(A, I) < 59, while Faltings’ result (3.1) shows that cd(A, I) < 60 — [59/5] = 49. If 
I is formally geometrically irreducible then (3.8) may be applied to obtain that 
cd(A, I) < 60 — [58/5] — 2 = 48. If in addition (A/I), is normal for all P = I with 
dim A/P = 3, then we can use (3.9) to get cd(A, I) < 60 — [60/6] — [59/6] = 41. 
Finally suppose that I, is set-theoretically generated by r elements for all primes 
P 21 with dim(A/P) 2 3, and I is formally geometrically irreducible. Then (3.10) 
applies to show that cd(A,1) < r+ 4, which is < 41 if r < 37. 


Remark 3.15. If we judge the efficacy of our estimates by the size of 
dim(A) — cd(A, 1), then our results show that when I is formally geometrically 


irreducible, we can do one better than in general, while if A/J is normal, we can do 
approximately twice as well. These results suggest that if one assumes that A/I 
satisfies Serre’s properties S; and R;_,, then better estimates can be obtained as 
i increases. However, as far as we can tell, our proofs can not be extended to prove 
any such result. 


Remark 3.16. This example shows that, at least in some cases, the bound given by 
(3.9) is best possible. Let X = (x;;) be a2 x n matrix of indeterminates. Let A be the 
polynomial ring in the 2n variables x;; localized at the irrelevant ideal. Set J equal 
to the ideal generated by the 2 x 2 minors of X. Then A/I is normal. In 3.13 we 
showed that cd(A,I)=2(n—2)+1=2n—3. We get from (3.9) that 
cd(A, 1) < 2n — 2 — [(2n — (n — 1) — 1)/n] — [(2n — (n — 1) — 2)/n] = 2n — 3. 
Hence the estimate is exact in this case. However, we have not in general found 
examples for all d and c showing that (3.9) is best possible. 


4. Prime ideals in power series rings 


In this section we prove several results about the extension of prime ideals in power 
series rings which were needed in Sect. 2. These results, particularly Corollary 4.4, 
were crucial to some of our theorems. In particular, Corollary 4.4 is needed to 
prove the vanishing theorems for normal ideals in characteristic p > 0. The main 
result is Theorem 4.2 which extends results in Chevalley’s paper [Ch], and Gilmer’s 
paper [G] (see also [Gr2, IV, 7.5]) and is of independent interest. It says that if 
a prime in a power series ring splits into several components after a field extension 
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of the coefficients, then it must already split in some finite separable subextension. 
Our first lemma shows this for polynomial rings—a result which is probably 
well-known, but for which we could find no reference. 


Lemma 4.1. Let P be a prime in R=k[x,,...,X,], where k is a field and R is 
a polynomial ring over k. Let L be an extension field of k, and setS = L[x,,...,X,]. 
If {Q,,...,Q,} are distinct minimal primes over PS, then there is a finite separable 
extension W of k, WC L, such that PW[x,,...,X,.] has t distinct minimal primes 
over it, namely Q;1W[x,,...,X,),1SiSt. 


Proof. Let F be the separable closure of k in L. First we show that PR, has 
t distinct minimal primes over it in R;, where we write Rg = G[x,,...,X,] for 
any extension ring G of k. It is enough to see that any prime Q in R, cannot have 
more than 1 minimal prime when extended to R,. Let N be the quotient field of 
R,/Q. Then R,/QR, =(R;/Q)@;L-— N @,L, so it suffices to see that every 
zero-divisor in N @,L is nilpotent. As F is separably algebraically closed in L, this 
follows at once from [Ja, Thm. 24, pg. 202]. 

Thus we may assume that L is separable algebraic to prove the lemma. In this 
case choose a generating set of each Q;. Adjoin all the coefficients of these 
polynomials to F, and call that field W. Note that W is a finite separable extension 
of F, and PRy must have t distinct minimal primes, namely, Q; = Q; 0 Ry. For 
Q; clearly is a prime containing PRy, and dim(Ry/Q;) = dim(R,/Q;) (by inte- 
grality) = dim(R/P), since Q; 0 R = P. Hence Q; is minimal over PRy, and they 
are necessarily distinct by construction. O 


We now will prove the same result for power series rings. First we establish 
some notation. Let R = k[[x,,...,x,]], where k is a field, and R is a power series 
ring over k. If G is any extension ring of k, let R, denote the ring G[[x,,..., x,]]. 
Let P be a prime in R, and by S denote R/P. By S, denote R,/PRg, if G is an 
extension of k. If a ring C is a domain, by F(C) we will denote the quotient field 
of C. 


Theorem 4.2. Adopt the notation of the above paragraph, and assume that L is a field 
extension of k. If PR, has exactly t distinct minimal primes Q,, . . . , Q, lying over it 
in R,, then there is a finite separable extension K of k, K S L, such that PRx has 
exactly t distinct minimal primes lying over it in Rx, which are the contractions of the 
Q;. In particular, there is a one-to-one correspondence between the minimal primes of 
PR, and PR,. 


Proof. Set d = dim(S). After a change of variables, if necessary, we may assume 
that A=k[[x,,...,x,]]<S, and S is a finite A-module. Let P’= 
POA[Xa+1,---5X,]. Note that A[xq+1,...,X,]/P’ maps onto S, has the same 
dimension, and is a domain; thus A[xg41,...,X,]/P’ = S. Observe also that 
P’AA=POA= (0). 

Let B= A, = L[[x,,...,X,]]. We claim that Q; 7 B = (0) for all i. Suppose 
not, set q,=Q,;B, with q,;+(0). We have that R,/PR, = R,/P’R, = 
BU Xa+1,---»5X,]/P’B[xXa41,...,X,] is integral over B, by the fact that the x,, 
with d + 1 Si <n, satisfy integral equations over A. Hence R,/Q; is integral over 
B/Q; 7 B, which in turn implies that dim(R,/Q;) = dim(B/q;) < dim(B) = d. How- 
ever, [Ch, Prop. 9] shows that dim(R,/Q;) = dim(R/P) = d. This contradiction 
shows that Q; 7 B = (0). 

As Q; \ B = (0) and P’ 4 A = (0), we must have that there are t minimal primes 
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over P’F(B)[Xa+1,...,X,], namely Q;0 F(B)[xa+1,...,%,]. By Lemma 4.1, 
this implies that the same is true in a finite separable extension W of F(A) 
contained in F(B). Consider the integral closure of A in W; call this C. Choose 
a primitive element ye F(B) which generates W over F(A). We can write y as 
a quotient f/g of two elements of C, and from elementary commutative algebra, we 
may assume that gé A. Since f is integral over A, we see that fe B, as B is integrally 
closed and fe F(B). Thus, f satisfies a separable integral equation over A, and 
W = F(A)(f). 

By [G, Thm. 4.1], the field K’ = k(coefficients of f) < L has the property that 
[K’: k ]sep < 00, and has finite exponent e over k. Let B = 4, where q = p*. Since 
W is separable over F(A), we have that W = F(A)(B). Let K be the field 
k(coefficients of 8). Then K is finite separable over k, and W c F(A,). (Note that 
this shows that fe F(A,), which in turn means that the coefficients of f are in K; 
thus K’ = K and K’ was actually separable. However, we shall not need this 
comment.) 

We claim that PR, has t distinct minimal primes over itself. It suffices to see 
that Q; 0 Rx are distinct (by flatness they are necessarily minimal over PR,), and 
for this it suffices to see that Q;7 F(Ag)[Xa+1,...,%,] are distinct. However, 
F(Ag) = K((x,,...,X,4)) contains W, and by the above argument, 
Q;,0 W[xa+1,--->X,] are distinct. Again by faithful flatness, every minimal prime 
of PR, is the contraction of one of the Q;, which finishes the proof. O 


Corollary 4.3. Adopt the notation of Theorem 4.2. Suppose that k is separably closed 
in L. Then nilrad(PR,) is a prime ideal. 


Proof. As k is separably closed in L, there is only one minimal prime above PR,, by 


Theorem 4.2. This gives the statement of the Corollary. O 


Corollary 4.4. Adopt the notation of Theorem 4.2. Assume that R/P is normal, and 
that L is any field extension of k. Then nilrad(PR,) is a prime ideal. 


Proof. By Theorem 4.2 it suffices to prove this corollary when L is a finite separable 
extension field of k. Set S = R/P. We claim that k is algebraically closed in F(S). If 
not, let K be the algebraic closure of k in F(S). Since K is integral over k, and S is 
integrally closed, K ¢ S, and hence K ¢ S/M, where M is the maximal ideal of S. 
Since S/M = k, we get a contradiction. 

By flatness, S @,L < F(S) @,L, so to prove the claim it suffices to see that 
F(S) @,L is a domain. However, since k is algebraically closed in F(S), and L is 
a separable extension of k, we may apply [Ja, Cor. 1, pg. 203] to obtain this 
conclusion. O 


Corollary 4.5. Let A and B be complete local rings containing a separably closed field 
k, which is the residue field of both A and B. Let Pe Spec(A), and Qe Spec(B). Then 
nilrad(P ®,B + A ®,Q) is a prime ideal in A ®,B. 


Proof. Both A and B are homomorphic images of power series rings 
over k; to prove the corollary we may assume A and B are equal to 
k[[x,,...,X4]], since the Corollary only depends on the quotients A/P 
and B/Q. Since k is a field, (A/P) ®,(B/Q) < (A/P) @, F(B/Q) = 
F(B/Q)(Lx,,.-.-,X4JJ/PF(B/Q) [[x,,...., xg]], so to prove the statement, it is 
enough to show that nilrad(F(B/Q){Lx,, ...,Xq]J/PF(B/Q)[Lx,,.--,X4]]) is 
prime. Since k is separably closed, Corollary 4.3 proves this. 
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5. Vanishing in regular local rings 


The purpose of this section is twofold. First we simply collect our results of the 
previous section in the case that the base ring is regular; this is for the convenience 
of readers. We give a result and examples to show that some of the results are the 
best possible. The main point of this section, however, is to prove a theorem which 
gives a sufficient (but not necessary) criterion for cd(A, 1) < d — 3, where A is 
a regular local ring of dimension d. Recall that in Theorem 2.9 we gave a new proof 
of theorems of Hartshorne and Ogus (in char. 0) and Peskine and Szpiro (in char. p) 
which gave necessary and sufficient conditions for cd(A, I) < d — 2. Our criterion 
in this section chiefly says that if every component of I has height at most (d — 2)/2, 
and no three minimal primes of J add to an m-primary ideal, then cd(A, I) < d — 3. 
This criterion allows us to give a complete answer to the question if the number of 
components of J is at most 5 and we assume bight(/) < (d — 2)/2. We begin, 
however, by collecting previous work. 


Theorem 5.1. Let A be an excellent regular local ring of dimension d containing 
a field, and I an ideal of A of bight(I) = b. Then: 


i) cd(A, 1) < d — [(d — 1)/b]. 
ii) If, moreover, I is formally geometrically irreducible, then 


cd(A, 1) Sd — 1 — [(d — 2)/b]. 


iii) If I is formally geometrically irreducible and (A/I)p is normal for all P 2 I 
such that dim(A/P) = 3 then. 


cd(A, 1) Sd — [d((b + 1)] — ((d — 1)(b + 1]. 


Proof. Statement i) is due to Faltings [Fal]. Statement ii) follows at once from 
(3.8), and iti) follows from (3.9). O 


Theorem 5.2. Let (A, m) be an excellent regular local ring of dimension d containing 
a field. Let b < d be a positive integer. Set s = [(d — 1)/b]. Let J,,..., J, be ideals 
of A of big height b such that Spec(A/(J, +...+ J,)) — {m} is disconnected. Set 
IT=J,0...A0J,. Then cd(A, 1) = d — [(d — 1)/b]. 


Proof. Set a; =(,<iJ,) + Ji4, +...+J,. We will prove by induction on i that 
cd(A, a;) = d — i. The theorem follows by setting i = s. If i = 1, the claim follows 
from (2.9). Assume that i> 1, and the claim has been shown for i— 1. Set 
J’=J,+...+J3, and J” =((\r<i-1J,) + Jian +... 4+J,. Then nilrad(a;) = 
nilrad(J’ J”), and a;-,; = J’ + J”. By the lemma of [Ly1], cd(A, J’) < d—i+ 1, 
and cd(A, J”) < d —i + 1. Now the claim and the theorem follow from the Mayer- 
Vietoris sequence for the ideals J” and J”, considering the induction hypo- 
thesis. O 


Corollary 5.3 [Ly1]. Let (A, m) be an excellent regular local ring containing a field. 
Let I,,...,1, be ideals of bigheight b such that 1, + ...+ 1, is primary to the 
maximal ideal. Set I = (); < 51 ;. Then cd(A, I) = d — s, where s is as in Theorem 5.2. 


Proof. Set J, = 1) 1,, and J; = 1; for i= 2. Apply Theorem 5.2. O 
Theorem 5.2 covers examples not covered by (5.3). Such an example is: 


Example 5.4. Let (A,m) be an excellent regular local ring containing a field. 
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Suppose P,,...,P, are = of height b such that P; + P, + P; is primary to 
the maximal ideal iff {i jk}eA = {(l, 3, 4), (1, 3, 5), (1, r 5), (2, 3, 4), (2, 4, 5)}. Set 
I=P,q...0P,. Note that A has the following property. Each i, 1 <i <5, 
appears no more than three times, and each pair {i,j} appears no more than twice. 
This implies that we cannot split {1, . . . , 5} into three disjoint groups C, D, and 
E in such a way that the sum of P; + P; + P, is m-primary whenever ie C, je D, and 
keE. Hence we cannot apply Corollary 5.3. However, we can split the ideal 
I = JK in such a way that Spec(A/(J + K)) — {m} is disconnected. Namely let 
J=P,OP,0P;,and K = P,P,. We leave it to the reader to prove this claim. 


Remark 5.5. The results of Theorem 5.1(i) and (ii) are exact for all d and b. Indeed, 
set s = [(d — 1)/b], and let Jo,...,J, be ideals of A of bigheight < b such that 
Jo+...+J, is primary to the maximal ideal of A. Set I= qJ;. Then 
cd(A, 1) = d — s. This follows from Corollary 5.3, and shows that (i) is the best 
possible. 


To show that ii) is the best possible for all b and d with 0 < b < d requires more 
work. We separate this in: 


Theorem—Example 5.6. Let k be an algebraically closed field. Set R= 
k[x,,.--,Xg-»], and set s = [(d — 1)/b]. Let J be the defining ideal in R of (s + 1) 
closed points in general position in A{~°. By W denote the (d — b)-dimensional 
affine variety whose coordinate ring is B = k @ J. Embed W in Aj for some n, and 
let W~ denote the closure of W in P{. Let V be the image of W~ after some generic 


projection of W~ on P{ in P%. Let A be the homogeneous coordinate ring of 
P? localized at the irrelevant ideal (of course dim(A) = d + 1). Finally, let I be the 
defining ideal of V in A. Then IJ is formally geometrically irreducible of height b, 
and cd(A, 1) = d — s = dim(A) — 1 — [(dim(A) — 2)/b], ie., the bound of (5.1) (ii) 
is exact for all such d and b. 


Proof. Clearly B is a domain. We claim B is a finitely generated k-algebra. Let 

1>---»M,4, be the maximal ideals of R such that J = 4 M,. Since R/M; = k 

each i, there exist cjek with x,—cj¢M,;. Then the elements 

s+1(x; — ¢j)€J, and R, and therefore B, is a finite module over 

., Ya-p]- Thus B is a finitely generated k-algebra of dimension d — b. By 

the same argument, R is integral over B, and is also clearly birational, with 

conductor J. It follows that B has an isolated singularity at the maximal ideal m of 
B generated by J, and that R is the integral closure of B. 

There exist exactly s+1 prime ideals in R lying over mR, namely 
M,,...,Ms+1. Hence the completion of B with respect to m (which we denote by 
B“) has s + 1 minimal prime ideals J,,... , Js+1. 

We claim that B“/J; is regular for each i, and that J; + J; is m primary if i + j. 
Indeed B%* has an isolated singularity at m, so nilrad(J; + J;) must be equal to m. 
The extension of B to R is unramified, and therefore so is the extension of 
(B*/J;) + (Ru,)*. As the residue field is algebraically closed, the extension is etale 
and must be an isomorphism. Thus B’/J; is regular, as claimed. 

Let G = k[[Z,,..., Z,,]] be the completion of the coordinate ring of Aj at the 
maximal ideal defining the closed point {m} of Spec(B). Let L < G be the extension 
to G of the defining ideal of W in Aj. From our calculations above, it follows that 
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L=L,n...0L,+;, where G/L; is regular of dimension d — b, and L; + L; is 
equal to the maximal ideal of G if i + j. 

Let Y,,..., Ya-, be d — b generic linear combinations of Z,,...,Z,. Then 
G/L; = k(LY1....,. Ya-»JJ, where the ’ means taking images in G/L;. After 
a change of variables, we may assume that Y; = Z;, for 1 Si<d—b. 

Let C=hk[[Z,,...,Z,]]} <G. We have that k[[Z,,...,Z4-,]]¢ 
CAL, AC)S G/L, = &{[Z,,...,Za-s]) which forces CAL; C) = 
kK{[Z,,...,Za-p,]]. Set N; =L,;0C. From linear algebra it is clear that if 
Zi-v+1,---,Z, are chosen generically, then the fact that L; + L; is to equal the 
maximal ideal of G forces N, +...+ N,+1 to equal the maximal ideal of C. 

Thus | < A is formally geometrically irreducible (since I is a homogeneous 
prime), but upon localizing and completing at a prime ideal of height d, it splits as 
in Remark 5.5. This shows that cd(A, 1) = d — s, as required. O 


We now prove a lemma needed in the proof of the main theorem. (see [Fa2] 
and [Br-R]). 


Lemma 5.7. Let (A, m) be a complete regular local ring containing its residue field k, 
which is separably closed. If P and Q are two prime ideals of A, with dim(A/P) 
+ dim(A/Q) > d + 1 = dim(A) + 1, then Spec(A/(P + Q)) — {m} is connected. 


Proof. Write A=k[[x,,...,X,]]. We may identify Spec(A/((P + Q)) with 
Spec((A/P) ®,(A/Q)/A), where 4 is the ideal generated by the diagonal in A ®, A. 
By Corollary 4.5, the ideal nilrad(P ®, A + A ®,Q) = Nin B = A @,A isa prime 
ideal, and dim(B/N) = dim(A/P) + dim(A/Q) 2 d + 2. Since d is the number 
of generators of A, by [Fa2, Thm.6], we get the desired conclusion, since 


Spec((A/P) ®,(A/Q)/A) — {m} is connected iff Spec(B/(N,A))—{m} is 
connected. (0 


Theorem 5.8. Let (A, m) be a complete regular local ring containing its residue field, 
which is separably closed. Let I be an ideal of A, with {P,,...,P,} the minimal 
primes of I, bight(I) = b. Then cd(A, 1) < d — 3 if 


i) 2b +2 <d and, 
ii) P; + P; + P, is not m-primary for any triple {i, j, k}. 


Proof. We may assume that J is reduced, hence J = 7 P;. We induct upon n to 
prove the theorem. First we consider the case n = 1. In this case, by Theorem 5.1 ii), 
cd(A, 1) < d —1 —[(d —2)/b] $d —3 as 2b+2<d. 

Let J = P,...0P,-,. By induction we may assume that cd(A, J) < d — 3. 
Consider the Mayer—Vietoris sequence for local cohomology with respect to 
J, P = P,,and I = JP. We obtain that H](M) = H4})(M) for all g => d — 2and 
for any A-module M, since cd(A,J)<d—3. Hence cd(A,1)<d—3 iff 
cd(A, J + P) < d—2. By Theorem 2.9, this is true iff dim(A/(J + P)) 2 2, and 
Spec(A/(J + P))—{m} is connected. First of all, dim(A(J+P))2 
dim(A) — ht(J) — ht(P) 2 2 by our assumptions. 

Suppose that Spec(A/(J + P)) — {m} is disconnected. Then there are ideals 
F and G such that F + Gis m-primary, FG © nilrad(J + P), but neither F nor G is 
contained in nilrad(J + P). Choose any P; for igSn—1. By Lemma 5.7, 
Spec( A/(P; + P)) — {m} is connected (since dim(A/P;) + dim(A/P) 2 2d — 2b 2 
d + 2). Hence for each i, either F or G is contained in nilrad(P; + P). If one of them, 
say F, is contained in nilrad(P; + P) for every i, 1 <i <n — 1, then F © nq (nilrad 
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(P; + P)) S nilrad( \(P; + P)) S nilrad(J + P). This contradiction shows that 
F Cnilrad(P; + P) for some i, and GC nilrad(P;+ P) for some j +i. Then 
m © nilrad(F + G) ¢ nilrad(P; + P; + P,), which contradicts ii). O 


Remark 5.9. Theorem 5.8 recovers the special case of a general ideal given in 
Theorem 5.1 i). For cd(A, 1) < d — [(d — 1)/b], shows that cd(A, I) < d — 3 pro- 
vided 3b < d — 1, a condition that forces condition ii) of Theorem 5.8 to be true 
since ht(P; + P; + P,) < 3b in general. Of course, (5.8) is more general than (5.1 i) 
since it only requires that 2b + 2 < d. One might expect similar better results for 
ideals of smaller codimension, but there are serious problems to extend the results 
further. Observe giving a necessary and sufficient criterion for cd(A, 1) < d — 3 is 
practically impossible. For let P be a prime with dim(A/P) = 3. Then if P is 
a set-theoretic complete intersection, cd(A, P) < d — 3, yet the vanishing of the 
obstructions to being a set-theoretic complete intersection are not sufficient to 
guarantee that cd(A,1) < d — 3, as they are for the inequality cd(A, J) < d — 2, 
where essentially being connected on the punctured spectrum guarantees the 
correct cohomological dimension. 


Remark 5.10. It is certainly possible to have some triples of minimal primes add up 
to be m-primary, but none-the-less have the estimate cd(A, I) < d — 3. For in- 
stance, the proof shows that at the inductive stage, not only must some triple add to 
be m-primary if the proof breaks down, but in fact one can divide the primes 
P,,..., P,-, into two disjoint sets, A = those containing F and IT = those con- 
taining G, such that given any P;e€ A, and any P;e/I, P; + P; + P,, is m-primary. 
Although this phenomena is suseptible to analysis, we have not yet found a definit- 
ive statement. 


To illustrate some of the problems with further analysis, a small amount of 
work yields a complete answer if the number of primes minimal over / is at most 5. 


Corollary 5.11. Let (A,m) be a complete regular local ring containing its residue 
field, which is separably closed. Let I be an ideal of A, with {P,,... , P,,} the minimal 
primes of I, bight(1) = b. Assume that 2b + 2 < d. 


i) Ifn <2, then cd(A, 1) Sd — 3. 

ii) If n = 3, then cd(A, 1) < d — 3 iff P, + P, + P; is not m-primary. 

iii) Ifn = 4 or 5, then cd(A, 1) < d — 3 iff whenever we divide {P,,..., P,,} into 
two disjoint sets C and D, and let J=()pecP; “aa K = ar 
Spec(A/(J + K)) — {m} is connected. Ifn = 4, then cd(A, 1) < d — 3 iff there 
is at most one triple {i,j,k} such that P; + P; + P, is m-primary. 


Proof. If n < 2, then the condition ii) of Theorem 5.8 is vacuously satisfied, so 
cd(A,1) < d — 3. Ifn = 3, and P, + P, + P; is not m-primary then we can again 
apply Theorem 5.8 to obtain that cd(A, 1) < d — 3. This condition is also necessary 
by 5.3 with s = 2. 

To prove (iii), first suppose that there exist two sets C and D as in the statement 
of (iii) such that Spec( A/((J + K)) — {m} is disconnected. Then H$; (A) + 0. Since 
H4~'(A) = H4-'(A) = 0 (for dimensional reasons neither Spec{A/J) — {m} nor 
Spec(A/K) — {m} can be disconnected, so we may apply (2.9)) it follows from the 
Mayer-Vietoris sequence that Hf *(A) + 0, so that cd(A, 1) = d — 2. Hence the 
stated condition is necessary. 

Assume the conditions in (iii). Certainly there must be some triple, say {1, 2, 3}, 
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such that P, + P, + P, is not m-primary (else the condition in (iii) is easily seen to 
fail). Set J = P,; 0 P,P, and K = P, ifn =4, and K = P,P, ifn =S. By (i) 
and (ii) of this corollary, cd(A, J) < d — 3, and cd(A, K) S d — 3. By assumption 
Spec(A/(J + K)) — {m} is connected, so that cd(A, J + K) Sd — 2 by (2.9). Now 
the Mayer-—Vietoris sequence shows that cd(A, I) < d — 3. The second statement of 
iii) for n = 4 is pure combinatorics and is left to the reader. O 


Example 5.12. Unfortunately, once the number of components of J is at least 6, the 
situation becomes much more complicated. A good case to consider is the follow- 
ing: let J be an ideal of bight(/) = b, with 2b + 2 < d, and suppose that J has six 
minimal primes, P,,..., P,. Set A = {{1, 2, 3}, {1, 3, 4}, {2, 4, 5}, {2, 4, 6}, {1, 5, 6}, 
{3, 5, 6}, {2, 3, 5}, {3, 4, 6}, {1, 2, 6}, {1, 4, 5}}. Suppose that {i,j,k}eA iff 
P, + P; + P, ism-primary. Is cd(A, 1) < d — 3? This example is arranged so that no 
matter how one divides I into two components J and K, either cd(A, J) or cd(A, K) 
is > d — 3. This prevents use of the Mayer—Vietoris sequence to compute cd(A, /). 


6. Topological applications 


In this section we present some topological applications of our vanishing theorems, 
applied mainly to subschemes of projective space. Most of the translation from 
cohomological information to topological information was done by Grothendieck, 
Hartshorne [Hal], [Ha2], and Ogus [O]. In particular we are able to show that 
for subvarieties of projective space, the homotopy groups are finite up to an integer 
depending upon the dimension of the ambient projective space, and the codimen- 
sion of the variety. We begin by proving a proposition which is probably well- 
known to experts. 


Proposition 6.1. Let X < P*(C) be a closed equidimensional algebraic set, and let 
Y ¢ X be aclosed subset of X such that X — Y is smooth. Let A be the homogeneous 
coordinate ring of X, and I ¢ A the defining ideal of Y. Then H,(X, Y; C) = 0 for 
i < dim(X) — cd(A, I). 


Proof. The local cohomology exact sequence implies that the cohomological 
dimension of Spec(A) — V(1) is cd(A, 1) — 1. Hence the cohomological dimension 
of X — Y is also cd(A, 1) — 1, since Spec(A) — V(I) is a C* bundle over X — Y. 
Now [Ha2, p. 148, Thm 7.4] implies that H‘(X; C) > H‘(Y; C) is an isomorphism if 
i < dim(X) — cd(A, I), and an injection if i = dim(X) — cd(A, I). By the universal 
coefficient theorem [Sp, pg. 244] this implies that the map H,( Y; C) > H,(X; C) is 
an isomorphism for i < dim(X ) — cd(A, I), and surjective if i = dim(X) — cd(A, I). 
This implies the result. O 


Theorem 6.2. Let Y < P*(C) be a closed algebraic set consisting of irreducible 
components of codimension < b. Then H,(P*(C), Y; C) = 0 provided, 


i) (cf. [P]) i S [e/b] — 1, if no additional assumptions on Y are made, 
ii) i < [(e — 1)/b], if Y is irreducible, 
iii) i< [(e + Ib + 1)] + [eb + 1)] — 1, if Y is normal. 


Proof. Let A be the homogeneous coordinate ring of P*(C), localized at the 
irrelevant ideal, and let J ¢ A be the defining ideal of Y. Then dim(A) = e + 1. We 
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see that c = c(I) = embdim(A) — mdim(A/I) < e — b. We may apply (3.8) to ob- 
tain that cd(A, I) < (e + 1) — [e/b], and hence by Proposition 6.1 the result (i) 
foilows. 

Consider (ii), using the same notation as in the proof of (i). Since I is 
formally geometrically irreducible, we can apply (3.8) to conclude that 
cd(A, I) < e + 1 — [(e — 1)/b]. Applying Proposition 6.1 yields (ii). 

Finally we prove (iii). Since J is formally geometrically irreducible, and A/I is 
normal on the punctured spectrum, we may apply (3.9) to obtain that 
cd(A, I) S (e + 1) — [(e + 1)(b + 1)] — [eb + 1)]. Applying Proposition 6.1 
gives (iii). O 
Remark 6.3. The result of (6.2 i) is exact for all e and b if b does not divide e. Let 
A be the homogeneous coordinate ring of P°(C), and let V < P*(C) be the union of 
[e/b] + 1 coordinate subspaces of codimension b in general position. Let J be the 
defining ideal of V in A. By (5.5) cd(A, I) = (e + 1) — [e/b]. By (5.1) i) for any prime 
P not equal to the irrelevant maximal ideal m, cd(Ap, Ip) < e + 1 — [e/b]. Thus 
H4(A) with q = e + 1 — [e/b] is supported at m only. It follows from the proof of 
[O, Thm 4.4] that in this situation H4(A) +0 is equivalent to the fact that 
H'"l(P°C) > H®"l( V; C) is not an isomorphism. Since this map is known to be 
injective, we conclude that H'@!(P*, V; C) + 0. By the universal coefficient the- 
orem, H,.)5;(P*°, V; C) + 0. In fact 6.21) is exact even if b divides e [Ly2]. 


Theorem 6.4. Let Y < P*°(C) be a closed algebraic set consisting of irreducible 
components of codimensions < b. Assume that Y can be cut out set-theoretically by 
t equations along every subvariety of P°(C) of dimension = m. Then H;(X, Y; C) = 0 


for 


i) i<e-—t—b+1lifm=1, 
ii) iSe—t—b+1,if Y is irreducible and m =2. 


Proof. The proof follows at once from (6.1) and (3.2) in case i), and from (6.1) and 
(3.10) in case ii). O 


Remark 6.5. Assume that Y is locally set-theoretically a complete intersection of 
codimension b except at finitely many closed points. From (6.4 i) we get that 
H,(X, Y;C) =0 for is e — 2b + 1. Similarly, if Y is irreducible and the set of 
closed points at which Y is not a set-theoretic complete intersection has dimen- 
sion < 1, we get from (6.4 ii) that H,(X, Y;C)=0 for i<e—2b +1. This is 
a generalization of the famous result of Barth [Ba]. 


Before we are able to apply these results to show finiteness of relative homotopy 
groups, we need to know that varieties of low codimension are simply connected. 
The results of Fulton and Lazersfeld allow one to show: 


Proposition 6.6. Let Y < P*(C) be a closed algebraic set consisting of irreducible 
components of codimension < e/3. Then Y is simply connected. 


Proof. Let Y,,..., Y, be the irreducible components of Y. We use induction on 
r to prove the proposition. If r = 1 the result follows from [F-L, Cor. 5.3, p. 47]. By 
induction Y’ = Y, nq... Y,-, is simply connected. The intersection Y’¢q Y, is 
the union of (r — 1) algebraic sets Y; 4 Y,, each one of which is connected by [F-H, 
Cor. 1], since codim( Y;) + codim( Y,) < e — 2. The intersection of any two of them, 
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say (Y;N Y,)A(¥,;9 Y,) = ¥,0 Y,- Y, is nonempty as the sum of the codimen- 
sions of any three of the Y, is at most e. Therefore Y’- Y, is connected, and the 
proposition follows from the Van Kampen theorem. CO 


Theorem 6.7. Under the hypotheses of (6.2) and (6.4), 7,(P°(C), Y) is finite in the 
same range as H,(X, Y; C) =0, except in (6.4 i) where we need the additional 
assumption that b < e/3. 


Proof. In (6.2 ii) and (6.4 iii), Y is simply connected by [F-L, Cor. 5.3, p. 47], if 
b < e/2, while in (6.4 i) it is simply connected by (6.6) (also in (6.2 i) provided 
b S e/3). Now (6.2) and (6.4) imply that H,(X, Y; Z) is finite in the same range where 
H;(X, Y; C) is zero, and since finite groups form a Serre class (see [Sp, Chap. 9, 
Sec. 6]), 2;(P°(C), Y) also are finite. 

If b2e/2 in (6.2 ii) and (64 ii, then i<1, and 2,(P°(C), Y= 
for Y connected, as can be seen from the exact sequence {1} + 2,(P*°(C), Y)> 
To( Y) 5 m9(P*°(C)). If e/2 = b > e/3 in (6.2 i), then i < 1, and z,(P*°(C), Y) = 0 for 
the same reason, since b < e/2 implies that Y is connected. Finally if b > e/2, in (6.2 
i) then i < 0 and 2,(P*(C), Y) =O forall Y O 


It is natural to conjecture that 2;(P°(C), Y) = 0 in the same range. This will be 
proved in an upcoming paper by the second author of this paper [Ly2]. 


Remark 6.8. Using (3.11) one can also prove the following result: 


Let X < P*(C) be an equidimensional closed algebraic set of dimension d, and 
let YX be a closed algebraic set consisting of irreducible components of 
dimension 2 d’. Assume that X — Y is smooth, and that the singular locus of 


X has dimension < d — d’. Then 


i) H,(X, Y; C) =0 for i Ss [(d + d’ — e)/(d — d’)). 

ii) In addition to the above hypotheses, assume that Y is ievedutible, and the 
singular locus of X has dimension < d — d’ + 1. Then, H,(X, Y; C) = 0 for 
i<[((d+d' —e-—1)(d—d’)} +1. 

iii) In addition to the assumptions of ii), assume that Y is normal. Then, 
H({X,Y;C)=0 for is[(d+d'-—e)(d-—d'+1)]}+[(d+d' —e-1)/ 
(d—d'+1)] +1. 
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Introduction 


Let K be an algebraic number field of degree d. Denote by M (K) the set of places of 
K and write M ,, (K) for the set of archimedean places of K. For ve M(K) denote 
by | |, the associated absolute value, normalized such that on Q we have 
| |, = | | (standard absolute value) if v is archimedean, whereas for v nonar- 
chimedean |p|, = p_' if v lies above the rational prime p. Writing d, for the local 
degree [K,:@,] we put || =| [t’4. 

Let S be a finite subset of M(K) containing M ,, (K) and having s elements. We 
call an element xe K an S-unit if 


|x |, = 1. for each v€S. (1.1) 
Let a,,...,a, be nonzero elements in K. We will study the equation 


aX, +... +4,x, =1 (1.2) 
in S-units x,,..., X,- 

Mahler [6] in 1933 proved that for n = 2 and for K = Q, Eq. (1.2) has only 
finitely many solutions in S-units. In his proof, he used his p-adic generalization of 
the Thue-Siegel theorem. For general n = 2 independently of each other Dubois 
and Rhin [1] and Schlickewei [9] showed in case K = Q that Eq. (1.2) has only 
finitely many solutions in S-units x,, .. . ,x, that are relatively prime in pairs. Later 
on independently of each other van der Poorten and Schlickewei [7] and Evertse 
[4] proved for an arbitrary number field K, that (1.2) admits only finitely many 
solutions in S-units xX,,...,X,, such that 

no proper subsum @;, X;, +... + %;,X;, vanishes . (1.3) 


1 th 


The example 2‘ — 2* + 1 = 1 shows that in fact (1.3) is also a necessary condition. 





* Current address: Department of Mathematics, University of Colorado Campus Box 426, 
Boulder, Colorado 80309-0426, USA 
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In both papers [7] as well as [4] the essential tool in the proof is the use of 
Schlickewei’s p-adic generalization [8] of W. Schmidt’s Subspace Theorem. As the 
underlying method in the Subspace Theorem is ineffective, it does not provide 
bounds for the sizes of the solutions x; of (1.2). However the situation changes when 
one asks for bounds for the number of solutions of (1.2). Evertse [3], using the 
theory of hypergeometric functions, proved that for n = 2 equation (1.2) has not 
more than 


3, 74+ 2s 


solutions in S-units x,,x,. For n22 Evertse and Gyoéry [5] showed, that the 
number of solutions of (1.2), (1.3) is bounded by a constant c that does not depend upon 
the coefficients a,,.. . , a,. Their method of proof however does not allow one to 
determine how the constant depends upon K, S and n. 

Recently W. Schmidt [14] succeeded in giving a quantitative version of his 
Subspace Theorem. This was generalized by Schlickewei [10] to include p-adic 
valuations. Using this result, Schlickewei [11] proved that for K = Q the number of 
solutions of (1.2), (1.3) is bounded by a constant c that depends only upon the 
dimension n and on the cardinality s of S. In [12] Schlickewei extended the 
quantitative Subspace Theorem to number fields. The main result of that paper 
allows us now to give an explicit upper bound for the number of solutions of (1.2) in 
general. 


Theorem 1.1. Let K be a number field of degree d. Let a,,...,a, be nonzero 
elements of K. Suppose S is a finite subset of M(K) of cardinality s, containing 
M.,,,(K). Then the equation 

Q,X,+ ...+4,x, =1 (1.4) 
has not more than 


(4sd!) 236n4's° (1.5) 


solutions in S-units x,,...,X, such that no proper subsum a,,x;, +... + 4;,Xi, 
vanishes. 


It should be pointed out that the 36 in the exponent of (1.5) could be certainly 
improved. We tried to avoid painstaking estimates. However, notice that the 
bound (1.5) is rather uniform in K and in S. It only depends upon the degree of 
K and upon the cardinality of S. In the opposite direction, it was shown by Erdés, 
Stewart and Tijdeman [2] that there exist equations (1.4) that have more than 
exp (Cs2/log s) solutions. 

In proving Theorem 1.1, it will be more convenient to consider the 


homogeneous S-unit equation corresponding to (1.4). In fact we shall prove that 
given nonzero elements a,,...,%,4, in K, the equation 


Xp. aX, + O41 X41 =O (1.6) 





has not more than (1.5) projective solutions (x,,..., X,41) satisfying the following 
properties. (x,,..., X,4 1) has a representative (x,,... , X,4,)€K"*! whose com- 
ponents x; are S-units and no proper subsum a;,x;, + . . . + %;,X;, Of (1.6) vanishes. It 


is clear that this claim is equivalent with the assertion of Theorem 1.1. 


Remark. It will be seen in the proof, that the d! in our bound (1.5) has its origin in 
the fact, that we make a detour via the smallest normal extension L of Q containing 
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K. For the degree of L we have the estimate [L: Q] < d!. If however K itself is 
a normal extension of Q, then it will follow from the proof below, that the terms d! 
in (1.5) may be replaced by d. (For details cf. Theorem 2.1 and Corollary 2.2 below.) 
Thus, if K is a normal extension of Q of degree d our bound (1.5) can be replaced by 


(4sd)?°°""s° : 


2. The subspace theorem 


The main ingredient in the proof is the quantitative version of the Subspace 
Theorem for number fields, which was proved recently by Schlickewei [12]. For the 
convenience of the reader this result will now be quoted. 

Given a vector a = (a,,...,)€K” we put for ve M(K) 


) a ; , 
(ja, |? +...+ a,|?)2 if v is archimedean 
|a|, = wi 
max |; |, if v is nonarchimedean . 


l<isn 


dy 
and ||@||, =|a|¥#. 
We define the height of a by 


H(a)= [] |lel,. (2.1) 
veM(K) 
Given a linear form L with coefficients «,,..., a, in K we put || L ||, = || @||, and 
H(L) = H(a), where a = (a,,..., &,). 


Theorem 2.1. Let L be a normal extension of Q of degree |. Let R be a finite subset of 
M(L) of cardinality r. Suppose that for each ve R we are given n linearly independent 
linear forms L{”,..., LS in n variables with coefficients in L. Let 0<6 <1. 
Consider the inequality 


mL (B) I, Be 
H one. 22 
HN joe pen <2 _ 


veR i=1 


There exist proper subspaces S,,...,S,, of L" with 
t, = [(ereyp""* 77, (2.3) 


such that every solution Be L” either lies in \_);_ , S; or satisfies 


H(p) < mine Quik. HL? 5 (i=1,...,mveER)} (2.4) 


Before we apply Theorem 2.1 to our problem, a few remarks seem to be 
appropriate. Initially we do not assume that our field K is a normal extension of Q. 
However let L be the smallest normal extension of Q containing K. Then 
[L:Q] =| <d!. We let R be the set of prime divisors we M(L) such that there 
exists ve S with w|v. Given veS, let w,, ..., w, be the extensions of v to L. Given 
a€ K it is well known that we have 


ol, = Woellw, --- lolly, - (2.5) 
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Therefore if we start out in Theorem 2.1 with K instead of L, with forms L{” with 
coefficients in K and with a subset S < M(K), we still may apply Theorem 2.1 with 
the set R as described above. Now R has cardinality < s(d — 1)! if S has cardinality 
s. Taking this into account in (2.3), the bound changes into 


[(8s(d — 198 dt? 180-2] < (asd)? 77 


Moreover the bound in (2.4) changes into 


9ns(d!)2 


9 
H() < max{(n!)6, H(L)?) > (i=1,..., mveS)}. 
We have proved. 


Corollary 2.2. Let K be a number field of degree d. Let S be a finite subset of M(K) of 
cardinality s. Suppose that for each ve M(K) we are given n linearly independent 
linear forms L®,..., L® in n variables with coefficients in K. Let 0<6 <1. 
Consider the inequality 


nL (B) Ie ie 
res ey 7 ee 2.6 
I rey, ren, <2 (2.6) 


There exist proper subspaces T,,..., T,, of K" with 
ty = [(deayrnr"*) (2.7) 


such that every solution Be K" either lies in the union \_);~ , T; or satisfies 


9ns(d!)2 


H(B) < max{(n!)5, H(L) 3s (i=1,...,n;veS)}. (2.8) 


It is this version of the Subspace Theorem we are going to use in the sequel. 


3. Application of the subspace theorem 


To apply the Subspace Theorem for the proof of Theorem 1.1, it will be convenient 
to make a change of variables. Put 


y,=a,x, (=1,...,n41). (3.1) 
Then Eq. (1.6) may be rewritten as 
Vit. ++ Yea, =9, (3.2) 


and we are asking for solutions y; of (3.2) satisfying 


Tile =T] lol. @=1...,041). (3.3) 
veS 


veS 


Write y =(y’, Yn+1) =(V1.--+ > Yao Ynti)- In the sequel, when we speak about 
solutions y, it is always meant that (3.2) and (3.3) are satisfied. 
Equation (3.2) defines a subspace say U of K"*! of dimension n. 


Lemma 3.1. There are 


t, =(n + 1)°.(4sd!)?°°*"“4s° 
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proper subspaces U,,..., U,, of U, such that for every solution y of (3.2), (3.3) the 
following holds true. 
Either y lies in the union § ae U; or we have 


H(y’) < max {(n!)!8, no"s ay? A2 otek | (3.5) 
where 


A=[[(hor i, -- ones I) [] Gmax {lay Il. sMtner Iod)"*? = (3.6) 
veS 


v¢éS 


Proof. We want to apply Corollary 2.2. For ve S we consider the linear forms 


C , 
LY (y y=y, 


< § 
LY (y’) = Yn 
L Lntaly’) = 94 +...+y,- 





Notice, that in (3.7) any n different among the n + 1 linear forms L(” are linearly 
independent. Moreover we have for i= 1,...,n and for any we M(K) 

WL ~e=1 and H(L{)=1 (3.8) 
whereas 


1 for nonarchimedean we M(K) 
A 


dw (3.9) 
n24 for archimedean we M(K) 


and therefore 
H(L®).,)=n2. (3.10) 
In view of (3.2) and (3.3) we get for any solution y 


LLC) ys tle s+ yea He) = TTI LP (yd I, - - Lay’) I) 
veS 


veS 


=[] (oy th... lone Ile) (3.11) 
veS 


To use Corollary 2.2, we now divide the solutions y into classes as follows. 
Given veS and y, define i(v) by 


| Yic le = MAX {]¥,|,--- 5 |Ynra ly} - (3.12) 


Given a tuple (i(v)),<5 with 1 < i(v) < n + 1, we denote by K = K ((i(v)),<s) the class 
of solutions y giving rise to the tuple (i(v)),-5 as in (3.12). The number of classes 
K equals 

(n+ 1). (3.13) 


We now fix one class K with corresponding tuple (i(v)),-5. Put I(v) = 
{1,..., + 1}\{i(v)}. Then (3.11) implies 


= % 
Il TT iz )te=(T] I| a le---ltes te) (TI I ue] . (3.14) 
veS 


veS iel(v) veS 





100 H.P. Schlickewei 


We infer from (3.1) and (3.3) that 
ly: = lo; ||, for any i(1 Sign -+ 1) and for any v¢S. 
Therefore we get 


[] max |yl,=[] max |lall,. 


v¢Sisisgn+l v¢éS lsisgn+l 


Combination of (3.14) and (3.15) yields 


LY ’ R —s~1 
ry te = a — Ite) ; 


veS ieI (v) max I Jj ll. veM (K) 1 Sisn+1 
l<jsn+1 


with A as given by (3.6). 
Given veS define k(v) by 


| Yew) |p = Max {lyil} - 
ieI (v) 


Then by (3.2) and (3.12) we obtain 


| Viwy |» = |Yew)|, for nonarchimedean veS , 
and 
| Velo S| Viele S21 Yew |, for archimedean veS . 
Therefore 
Yi lp = lly’, for nonarchimedean veS , 


and 
d 


v ldy 
n-@ | yi ll, S iy’ ll, $n? || Yigy |], for archimedean veS . 


Using (3.8), (3.9), (3.18), (3.19) we infer from (3.16) that 


| L(y’) Il, 1 ae 
. < An2" H( ’) n-1 . 
I I ILI, Vy’ I. , 


Assume that 
H(y’) > Az no"*2 : 


then (3.20) implies (remember (3.10)) 
| LY (y’) Il, Sone. 
oe << (iy) (3.22 
Nl aeaiy <2 ) 


We are now in a position to apply Corollary 2.2 with 6 = 4 and we may infer 
that there exist : 
t, = [(4sd!)?°"* 48°] (3.23) 


subspaces V,,..., V,, of K” containing all solutions y’ of (3.20) satisfying 
H (y’) = max {(n!)'8, n°") (3.24) 


Summing up over the different classes K introduces the factor (n + 1)° from (3.13), 
and the Lemma follows from (3.23), (3.21) and (3.24). 
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4. A gap principle 


Lemma 4.1. Suppose 0 < y < 1 and qéN are given. Then there exists a subset M of 


cardinality : 


er 
of the set {(,,...,T,)|[, + ...+ 1, =7,0;20(=1,..., q)} with the follow- 
ing property. For every point x =(x,,...,X,) in R* having x;20 for each 
i (1 Si<q), there exists a point [=(I,,...,1,)€M such that for each 
i(l Si<q) 
% HT Oy +... RD 
holds true. 


This is Lemma 5.1 of [12]. As mentioned there, it appears implicitly already in 
Mahler [6]. 


Lemma 4.2. Let B be a quantity with 
Bana” . 
Put 


1 
C= Az (n?=1) 


with A as in (3.6). 
Then the solutions y of (3.2), (3.3) with 


Bs H(y') SBC (4.3) 
are contained in the union of not more than 


(n + 1)?52?""s (4.4) 
proper subspaces of U. 


Proof. Given n solutions y, 29M of (3.2), (3.3), (4.3), we shall consider the matrix 


Vito+++oVi,nt1 


(4.5) 


Yni> Prats »Yn,n+1 


where y;; is the j-th component of y;. Denote by D; the determinant of the matrix 
obtained from (4.5) by deleting the j-th column. Then (3.2) implies 


D,;=D, for eachj(l Sj sn+1). (4.6) 
It was shown in (3.11) that any solution y of (3.2), (3.3) satisfies 
TLC ya de. + ll ynea He) = TT hoy Wl, - Monet Ul) - (4.7) 


veS veS 


Let K = K ((i(v)),<5) the class of solutions y defined by (3.12). Remember the 
definition of k(v) in (3.18). Each class K splits into 


n* (4.8) 


subclasses K’, where a subclass K’ consists of those y in K that give rise to a fixed 
tuple (k(v))es- 
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Fix a subclass K’ and put for veS J(v) = I(v)\{k(v)} with I(v) as in Sect. 3. 
Then combining (4.7) with (3.20) we get in view of (3.3) for each y in K’ 


ly: ll, < nag 
nal Fm ty) 
veS ies(v) MAX [Yj I, veM (K) 1Sj<n+1 

l<jsnt+l 


We observe that by (3.19), (3.20) this implies 
Y n+3 
I Ji ll. < n2 AH(y’)"""! : (4.9) 

veS ieJ (v) max I Jj ll» 
lsjsn+l 
In (4.9) each factor of the double product op the left hand side is bounded 
by 1. Therefore, there exist nonnegative real numbers x,,(veS, ie J(v)) with 
YY x, = 1 and with 


veS ieJ (v) 
Il vill, (1 Il | (veS, ie J(v)) . 
veS keJ (v) ; 


max |ly;ll,_ max || y; ll, 
isjgn+l Sjgn+l 
We may apply Lemma 4.1 with 0 < y < 1 yet to be specified and with q = s(n — 1). 
Consequently and in view of (4.9) there exists a tuple (7;,) with veS, ie J(v) in 
M having 
y ri, =, rj, 20 
veSieJ(v) 
and 


ll Yille 


I Yel y" (n? . y" 
silane et Sin * AMfy’)-e*” (4.10) 
max || y; lk. (10 max [| y; lle y) 
1 j 1 


Isjsgn+l Sjgn+ 


for each pair (i, v) with ve S, ieJ (v). 
Recall from Lemma 4.1 that the set M has cardinality 
*s(n-1) 


1 
<2'-7 (4.11) 


We subdivide K’ further into subclasses K” = K" ((I;,,,)). K” consists of those 
yéK’ that satisfy (4.10) for the fixed tuple (Jj). We proceed to show that any 
n solutions y,,..., y, that lie in a fixed subclass K” are linearly dependent. 

For this purpose we will estimate the (n x n)-subdeterminants of (4.5). We may 
assume that y,,..., y, are ordered such that 


H(y;)S H(y2)S...S H(y,). (4.12) 
Then (4.10) implies 





lI Yai ll s(n? aniy,y-*?) " (4.13) 
max [lysjlly 


Isjgn+l 
for each k(1 < k <n) and for each pair (i, v) with veS, ie J (v). 


For veS choose |e {1,..., + 1}, 1,¢J(v) and consider the determinant D,,. 
Using (4.13) we obtain 


n+3 ~s Ty. on 
ID. es (n Any.) T] max |yjll, (4.14) 


k=11<jsn+1 
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for nonarchimedean ve S and 


dy ( "+3 dln 
|| Di, Il, sin) (n Anyi)? )* [] max |y,; ll, 


k=11<Sjsn+1 


for archimedean ve S. 
Combining (4.14), (4.15) and (4.6) we see that 


a 2 
Ds te snt(n . 4) H(y,)°* [] [J] max i ygjll.- 


veS veS k=1 1 Sjsn+l 


Now suppose v¢S. Here we have by (3.1), (3.3) 


Iv, = lol, - 
Choose /,, such that || % \|,= max ||; ||, . Then we get by (4.17) 
Ilsign+l 
n+1 


ID. oS I] dole (4.18) 
i=1 


i+ly 


Using the relation [] max || y,;||, = ( max |/¢; .) , we infer from (4.18) 


k=11sjgn+1 Isjsn+l 


n+1 


Hite "4 





k= lIsjs 1 
max ian 


Isjgn+l 


oa ta} n+1 I] max II Vij lle « 
|| a l) 


Taking the product over v€S, observing (4.6) and using the fact that 


1 i 12; -(1T Il tat) snail 


veS i= v¢S i= 
n 
[] I] max II Yej lle 
[| \D | < véSk=11SjSnt+l 
v= 


, - iat.) max lay.) 


veS i=1 v¢Sisjsn+i 





wh) i max ll Yej lle - 
v¢éS k= 


Isjgn+l 


The bounds in (4.16) and in (4.19) give together 


n+3 


I] WD, il,gatn? Amt ayy rt TT TT max | ysl. 


veM(K) veM(K) k=1 1sjsn+1 
With (3.19), (3.20) we conclude that 


n+3 


I] WD, I, Salm? Arm? Alyy) rn" Hl H (yx) 


veM(K) k= 


<i sey Il H(yx) - 


k=1 





104 H.P. Schlickewei 


We now choose 
2n + 1 


ae CEs 
Then (4.20) implies in conjunction with (4.2), (4.3) 


1 1 1 
1 18 A,<e" A ey) ° B.A” 


veM(K) 


1 
=n H(yi)"2. 


Since we may assume that n 2 2, (4.1), (4.3) yield 


I] ID,i.<1. 
veM(K) 
Therefore D, = 0 and by (4.6) y,,...,y, are linearly dependent. 

We have shown that the solutions y in a subclass K” defined via the tuple (T;,,) 
and the subscripts i(v), k(v) are contained in an (n — 1)-dimensional subspace of the 
solution space U. 

By (3.13), (4.8) and (4.11) the total number of subclasses is bounded by 


(n a 1) n° 22 (n? — 1s 


and the Lemma follows. 


5. Small solutions 


Lemma 5.1. Let D = 1. Then the number of one-dimensional subspaces in K" having 
a basis vector x which satisfies 


H(x) < D (5.1) 
is bounded b 
‘ n36"~! J24(n— 1) py2d(n— 1) : (5.2) 


Proof. This is a consequence of Lemma 8B of W.M. Schmidt [14]. In fact, it suffices 
to count the number of points x = (x,,..., x,) with at least on component equal 
to 1. Suppose x = (1, x,,..., X,). Then 


H(x)= max H((1, x;)). (5.3) 
2sign 


Schmidt’s Lemma says that the number of elements xe K having H ((1, x)) S D is 
bounded by 


36(2D)*4 . 


Allowing n — 1 in the exponent for the number of possible choices of the maximum 
in (5.3) and the factor n for the number of choices of the component that equals 1, 
the assertion follows. 
Applying Lemma 5.1 with 
Diss no"s (d!)2 
we get at once. 
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Lemma 5.2. The solutions y of (3.2), (3.3) with 
H(y') < nome? (5.4) 
are contained in the union of not more than 
ts = n 36°27 newt? (5.5) 
proper subspaces of the solution space U. 
The remaining solutions will be taken care of by means of the gap principle. 
Lemma 5.3. The solutions y of (3.2), (3.3) with 
n°” < H(y') < A2n3"*? (5.6) 
with A as in (3.6) are contained in the union of not more than 
te = 4(n? — 1)(n + 1)752?"" (5.7) 
proper subspaces of the solution space U. 
Proof. By Lemma 4.2, for each subinterval of the interval (5.6) of the shape 


[e.0r ME) 
(n + | 22n?s 


subspaces will suffice. On the other hand the interval (5.6) may be covered by 
4(n? — 1) such subintervals and the assertion follows. 


6. The total number of subspaces 


Lemma 6.1. Let U be the n-dimensional subspace of K"*' defined by Eq. (1.6). 
There are 
t, = 2(n + 1)*(4sd!)?°""* 4"° (6.1) 


proper subspaces V,,...,V,, of U such that every solution x of (1.6) whose 


components are S-units is contained in the union \ );_, V;. 


Proof. It suffices to add the numbers t,, t;, t, from Lemmata 3.1, 5.2, 5.3, since the 
bounds given in (3.5), (5.4), (5.6) complement each other. We obtain 


2 3 4n2 
ae tei" 
and 


t3 +s +t, < 2t, = 2(n + 1)' (4sd!)?°°"“"4*° , 


7. Proof of the theorem 


In this section, when we speak about solutions of (1.6), it is always tacitly assumed 
that the components are S-units. 

We proceed by induction on n. Let Z(n) = Z(n, d, s) be the constant in (1.5). We 
want to show that Z(n) is an upper bound for the supremum over @,,... ,@,4, Of 
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the number of projective solutions of (1.6). It is clear that for n = 1 there is at most 
one projective solution and our bound in that case is more than generous. 

We now assume n > | and the Theorem to be proved for all n’ < n. Let V be 
one of the subspaces of Lemma 6.1. We shall first only study solutions x of (1.6) 
lying in V. Since V is a proper subspace of U, the solutions x ¢ V satisfy apart from 


(1.6) an equation 
Y Bx; = 0 (7.1) 
iel 


with noazero coefficients B;¢ K and a subset J of {1, .. . ,n} of cardinality | J| = 2. 
Let J be a nonempty subset of J and consider those solutions x for which 


> Bx; = 0 (7.2) 


ieJ 
but no proper subsum of (7.2) vanishes. Notice that we have 
|J|>2. (7.3) 


Since | J | < n we may apply the induction hypothesis to (7.2) and we see that (7.2) 
admits not more than Z(|J| — 1) projective solutions. Now for each solution x of 
(1.6) with xe V and with (7.2), the components x; with ieJ are of the shape 
(x; )iesy = (X7;)ies, Where (y;)icy is one of the projective solutions of (7.2). Moreover 
x is an S-unit. Using this in (1.6) we obtain 


n+1 
( 7. ain) + a,x,=0. (7.4) 
=1 


ieJ 


i 
i¢J 


By assumption, in (7.4) no subsum vanishes. Therefore the induction hypothesis 
says, that (7.4) admits not more than Z(n + 1 —|J|) projective solutions with 
components say 6, 6; (i¢ J). Hence the points with components dy; for ie J and 6; 
for i¢ J represent all projective solutions corresponding to our set J. Thus there are 
not more than 


Z(\J|—1IZ+1—-|J|)SZ(n—- 1? 


projective solutions in V arising from the set J. 

The number of possibilities for our subset J is bounded by 2”. Therefore V does 
not contain more than 2” Z(n — 1)? projective solutions. Using the bound (6.1) for 
the number of subspaces we see that (1.6) has not more than 


2"Z(n — 1) t, 
projective solutions. We obtain 
2"Z(n ag 1)?t, $ 2" (4sd! )? 36(n— 1)d's®2 2(n ob 1)°(4sd!)?°°"*- 48° 
< (4sd!)? 36(m— 1)4'28° (4 cqny2?% n°. 4s6 
< (4sd!)?°°"“5* = Z(n) . 
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In this paper we want to discuss the following 


Conjecture. A compact non-singular algebraic surface with finite fundamental 
group has at least two smooth structures which are stable under blow ups. 


This means that for each compact algebraic surface X there exists a homeomor- 


phic smooth manifold Y such that X # s-C P? is not diffeomorphic to Y #s-CP? 
for all s = 0. To consider smooth structures which are stable under blow ups is very 
natural from the point of view of algebraic geometry since one has only to study 
minimal surfaces. Note that the conjecture is false if one allows in addition 
(arbitrary) stabilization with C P? ([W] in the 1-connected case and in general [G] 
or [Kr, ]). 

The main result of this paper is the following. 


Theorem. Let G be a finite group. Then there is a constant c(G) such that the 
conjecture holds for all algebraic surfaces X with n,(X) = G, Euler characteristic 
e(X) = c(G) and c?(X) = 0. 


The condition c? > 0 holds automatically for minimal surfaces with finite 
fundamental group ([BPV], p. 188). Note that for each finite group G there exist 
algebraic surfaces X with 2,(X) = G, c?(X) = 0 and arbitrarily large Euler charac- 
teristic. Let Y be a minimal model of an algebraic surface with fundamental group 
the symmetric group S,, [Sh, p. 402 ff ]. Embed G into S,, for some m and let X be 
the corresponding covering over Y with 2,(X) = G. As noted above c?(Y) >0 
and thus c?(X)>0 and we can make e(X) arbitrarily large by choosing m 
appropriately. 

In (THK, ], Corollary 1.5), we proved that there are only finitely many homeo- 
morphism types of closed oriented 4-manifolds with given finite fundamental group 
G and fixed Euler characteristic. 

Thus we obtain: 


Corollary. Let G be a finite group. Then the conjecture holds for all but perhaps 
a finite number of homeomorphism types of minimal algebraic surfaces X with 
1,(X) =G. 
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Our conjecture and the partial results should be seen in the context of a conjec- 
ture of Friedman and Morgan which states that the natural map from algebraic 
surfaces modulo deformation equivalence to smooth 4-manifolds modulo orienta- 
tion preserving diffeomorphism is finite-to-one [FM, ]. The proof of this conjec- 
ture was recently announced in [FM, ]. This result solves our problem in some 
important cases like for instance the K3-surfaces. 

Our conjecture is also related to a conjecture of A. Van de Ven: algebraic 
surfaces of different Kodaira dimension are never diffeomorphic. If this is true, our 
conjecture holds for those algebraic surfaces which are homeomorphic to an 
algebraic surface of different Kodaira dimension. An interesting test case for Van 
de Ven’s conjecture is the Barlow surface X which is homeomorphic_ to 
Y = CP? # 8-CP?, but is of general type. Recently it was shown that X # s-C P? 
is not diffeomorphic to Y # s-C P? for all s => 0 [Ko]. 

In Sect. 1 we summarize some known methods and results concerning the 
conjecture and in Sects. 2 and 3 we give the proof of our Theorem. 


Acknowledgements. The second author would like to thank the I.H.E.S. for its hospitality and 
support. Both authors were partially supported by NSERC grant A4000 and the University of 
Mainz. 


1. Background 


In this section we summarize some facts and results concerning the conjecture for 
specific fundamental groups. 

The topological s-cobordism Theorem [Fr,] reduces the homeomorphism 
classification for nice fundamental groups to finding topological s-cobordisms. On 
the other hand Donaldson introduced new diffeomorphism invariants ([Do, ], 
[Do, ]) which are closely related to algebraic geometry. 

In particular Donaldson [Do,] shows that a simply connected compact 
algebraic surface is essentially indecomposable. 


Theorem 1.1 [Do,]. Let X be a 1-connected compact algebraic surface. Then X is 
not diffeomorphic to a connected sum M, # M, unless M, or M, have negative 
definite intersection form. 


One can use this result in combination with Freedman’s classification of 
1-connected manifolds to prove the conjecture for 1-connected algebraic surfaces 
which are homeomorphic to decomposable smooth manifolds. 

This shows that the conjecture holds for all 1-connected compact algebraic 
surfaces X which are not homeomorphic to K, CP! x CP! or CP? #s-CP?. 
A more detailed analysis of the Donaldson invariants leads to a proof of the 
conjecture for CP? #9-CP? [FM,], the Kummer surface [FM,] as well as 
CP? # 8-CP? [Ko]. 

We summarize the present status of the conjecture for 1-connected surfaces as 
follows. 


Theorem 1.2 ([Do, ], [Do,], [Fr, ], [FM,], [FM;], [Ko]). The conjecture holds 
for compact 1-connected algebraic surfaces which are not homeomorphic to 
CP! x CP! or CP? #s-CP? and s <7. 





Smooth structure on algebraic surfaces with finite fundamental group 111 


In the non simply-connected case more work must be done on the homeo- 
morphism classification to apply these ideas. 


Theorem 1.3 (CHK, ], [T]). The conjecture holds for compact algebraic surfaces 
X in the following cases: 


i) 1,(X) = Z,,k > 1 and odd 
ii) 1,(X) = Z,, and e(X) +4 
iii) m,(X) = SL,(F,), p = 3 or 5 (mod 8), c?(X) = 0 and e(X) > 12 


Remark. From the known results concerning our conjecture one gets the impres- 
sion that the simpler the topology of a surface the harder the problem. The hardest 
case seems to be CP”. For CP? # CP? one has at least a candidate. Note that 
CP? # CP?” is the 2-fold branched covering of S* along the standard embedding of 
RP? # RP? with normal Euler class 0 (with twisted coefficients). In [FKV] an 
infinite family of new knottings is constructed. By [Fr, ] each ramified covering is 
homeomorphic to C P? # C P?. The same construction can be carried out starting 
from # 2r-RP?GS*. Ifr = 5 the corresponding ramified covering is diffeomorphic 
to a Dolgachev surface which is not diffeomorphic to C P? # 9-C P?. It would be 
somewhat surprising if the analogous family of knottings for r = 1 would lead to 
diffeomorphic ramified coverings. 


In the situation of Theorems 1.2 and 1.3 the problem simplifies if one blows up 
an algebraic surface a few times. It is perhaps more realistic to study the following 
weakening of our conjecture. 


Weak conjecture. For each algebraic surface with finite fundamental group there 
exists an r such that after r blow ups it has at least two smooth structures which are 
stable under further blow ups. 


Theorems 1.2 and 1.3 imply the weak conjecture for all algebraic surfaces with 
finite cyclic fundamental group. 


2. Proof of the theorem 


Two 4-manifolds M, and M, are called stably homeomorphic if for some natural 
numbers r, andr,, My # ro(S? x S*)is homeomorphic to M, # r,(S? x S?). Note 
that ro =r, if and only if e(M,) = e(M,). In [HK,] we proved the following 
cancellation result 


Theorem 2.1 ((HK.,; 1.3]). Let M, and M, be stably homeomorphic closed smooth 
4-manifolds with the same finite fundamental group and Euler characteristic. Suppose 
that M, is homeomorphic to Mo # 2(S? x S?). Then Mg is homeomorphic to M,. 


We define a Z-action on the set of stable homeomorphism classes of smooth 
closed oriented 4-manifolds with fixed fundamental group G by connected sum 
with r copies of the Kummer surface K:(r, M)++M #r-K. Here for r <0, rK 
means |r|-( — K). Note that K # (— K) is homeomorphic to # 22(S? x S?). The 
next step in the proof is the following finiteness result which we will prove in the 
next section. 
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Proposition 2.2. Let G be a finite group. The set of orbits of stable homeomorphism 
classes of smooth closed oriented 4-manifolds with fundamental group G under the 
action given by connected sum with K is finite. 


For each orbit choose a representative M; with e(M;) minimal such that 
—8 <signM, <8, and M, = M; # 2(S? x S?). Then for each closed oriented 
smooth 4-manifold X with 2,(X) = G there exist i and r such that X is stably 
homeomorphic to M; # r- K. If e(X) 2 e(M; # r: K) then X is homeomorphic to 
M, #r-K # s(S? x S?) = Y for some s by Theorem 2.1. On the other hand 
Donaldson’s Theorem (1.1) implies that if X is an algebraic surface, X and Y are 
not diffeomorphic (note that M; decomposes as M; # 2(S? x S”). Thus the proof of 
our Theorem is finished if we can find a number c(G) such that e(X) = e(M; # r- K) 
for any algebraic surface X with n,(X) = G, e(X) = c(G) and c7(X) 2 0. 

To compare e(X) with e(M, # r- K) we express e(M; # r- K) in terms of e(M,), 
sign(M;) and sign(X): 

e(M; # r-K) = e(M,j) + 22-|r| 
and 
sign(X) = sign(M,; # r: K) = sign M; — 16r. 
This implies 
e(M; #r-K) = e(M,) + (11/8)|sign(X) — sign(M;)| . 
Assume now that X is not simply-connected (see (1.2)). We have the following 
inequality: 
e(X) — (11/8)|sign(X)| 2 (1/12)e(X) . 
If sign(X ) < 0 this is an immediate consequence of the signature theorem and the 
assumption c?(X) = 0. If sign(X) = 0 then c?7(X)>0 and X is of general type 
([BPV], p. 188). We now apply the Miyaoka-Yau inequality [BPV, p. 207]. 


This implies together with the formula for e(M; #r-K) above (note that 
sign(X ) = sign(M;) mod 16 and — 8 < sign(M,;) < 8): 


e(X) — e(M; # rr: K) = e(X) — (11/8)|sign(X)| — e(M;) + (11/8) sign(M;) 
2 (1/12)e(X) — e(M;) — 11. 


Thus, if e(X) = 12 e(M;) + 132 we have e(X) = e(M; #r- K). 
As there are only finitely many M,’s we can define 


c(G): = 12- max {e(M;) + 11} 


finishing the proof of our Theorem. 


3. Proof of proposition 2.2 


The normal 1-type of a closed smooth oriented 4-manifold M is the fibre homotopy 
type of a fibration ¢: B + BO which is completely characterised by two facts: the 
homotopy groups of the fibre z,(F ) vanish for r = 2 and there is a lift of the normal 
Gauss map v:M > BO to B, denoted by ¥:M-—B, which is a 2-equivalence 
(isomorphism on 7, and surjective on 2,, [Kr,]). In other words, the normal 
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1-type is the first stage of a Moore—Postnikov decomposition of the normal Gauss 
map. The pair (M, v) represents an element in 2,(¢), the bordism group of oriented 
4-dimensional €-manifolds. The group of fibre homotopy self-equivalences Aut(é) 
operates on 2,(€) by composition and the class represented by (M, v) in 2,(€)/ 
Aut(€) is independent of the choice of v[Kr,; p. 16]. The set of stable homeomor- 
phism classes of smooth oriented closed 4-manifolds M having € as its normal 
1-type is isomorphic to 2,(€)/Aut(é) [Kr,; 2.4]. 

The bordism group 2,(&) is finitely generated and 24) @Q= ~ H,(B; Q) 
Pay [M,v]rv «LM ] (compare eS As 7,(B) is_ finite the transfer 

H,(B; Q) > H,(B; Q) is injective. Also B is BSO ‘f w2(M) +0 and BSpin if 
w,(M) = 0. Thus 2,(&) @ Q = Q and the isomorphism is given by the signature. 

Since the Kummer surface is 1-connected and almost parallelizable, connected 
sum with K does not change the normal 1-type of M. As sign(K) + 0 we see that 
the orbit space of stable diffeomorphism classes of closed smooth oriented 
4-manifolds M with normal 1-type € under the action of K is a finite set. 

The proof of Proposition 2.2 follows from this if we show that for a fixed finite 
fundamental group G the number of normal 1-types of manifolds M with 

1,(M) = G is finite. 
This is obvious if w,(M) +0 since then the normal 1I-type is k(z,(M), 1) 


x BSO — > BSO. If w,(M) = 0, let f: Y-> M be a map inducing an isomorphism 
on 7,. Then it follows immediately from the characterization of a Moore—Pos- 
tnikov factorization that the normal 1-type is determined by (Y, vof). We can 
obtain Y as the geometric realization of a fixed presentation of G = 2,(M). But the 
set of homotopy classes of maps Y — BO is finite. 
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Summary. The main result concerns changing an arbitrary closed braid represent- 
ative of a split or composite link to one which is obviously recognizable as being 
split or composite. Exchange moves are introduced; they change the conjugacy 
class of a closed braid without changing its link type or its braid index. A closed 
braid representative of a composite (respectively split) link is composite (split) if 
there is a 2-sphere which realizes the connected sum decomposition (splitting) and 
meets the braid axis in 2 points. It is proved that exchange moves are the only 
obstruction to representing composite or split links by composite or split closed 
braids. A special version of these theorems holds for 3 and 4 braids, answering a 
question of H. Morton. As an immediate Corollary, it follows that braid index is 
additive (resp. additive minus 1) under disjoint union (resp. connected sum). 


1. Introduction 


This paper is part of a series of papers ([B-M, I, II, III, ['V and V] in which the 
authors study representations of oriented links in oriented S? by closed braids. The 
long-term goal is to produce computable link type invariants by making use of one 
of the known solutions to the conjugacy problem in the braid group (e.g. [G]). Any 
such effort must deal with the various mechanisms which produce more than one 
conjugacy class of braids in a braid group B, which represent the same link type. 
Two such mechanisms are treated in this paper. 

A link K is split if there is a 2-sphere X in S?-K which does not bound a 3-ball. 
A closed braid representative K of a split link K is a split closed braid if the splitting 
2-sphere X can be chosen to meet the braid axis A in exactly two points. The closed 
4-braid a, in Fig. 1 is an example of a split closed 4-braid which represents the 
disjoint union of two trefoil knots. 

Since every link can be represented by a closed braid, it is obvious that every 
split link can be represented by a split closed braid. For, if K is the disjoint union of 
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split closed braid non-split closed braid split closed braid 


Fig. 1. Closed 4-braid representatives of the disjoint union of two trefoils 


links K, and K,, choose an n,-braid representative K, of K, and an n,-braid 
representative K, of K, and use them to construct the obvious (n, + n,)-braid 
representative K by banding together their respective axes. There is, however, a 
more subtle question, i.e. are there n-braid representatives of split links which are 
not isotopic in the complement of the axis to split closed n-braids? 

The answer is “yes”, and an example is given by the closed 4-braid «, in Fig. 1, 


which also represents the disjoint union K of two trefoil knots. If «, were isotopic in 
the complement of the axis to a split 4-braid representative of K, it would 
necessarily be conjugate to «, because there is a unique closed 2-braid which 
represents the trefoil, and so a unique split closed 4-braid which represents the 


66299 
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disjoint union of two trefoils. Let the symbol “i” (resp. “i~ !”) denote an elementary 
braid in which the i" strand crosses once over (under) the (i + 1)* strand. Then a, 
and a, may be described by the braid words: 


ay = 133° 9 
a, = 122723322, 


Now, there is a homomorphism ¢: B, > B, defined by $(1) = (3) = 1, @(2) = 2. 
If ~, and a, were conjugate in B, then ¢(a,) would be conjugate to @(a,) in B3. 
Using the solution to the conjugacy problem in B, which is given in [Mu] we 
obtain unique representatives 1° for the conjugacy class of @(a,) and 
A?1?2~?172~? for the class of (a2). Thus a, and «, are not conjugate in B,. By a 
theorem of Morton [Mo, 3] this shows that the corresponding closed braids are 
not isotopic in the complement of A. Infinitely many such examples can be 
obtained by iterating the winding process which is shown in Fig. 1. 


Remark. The example a, in Fig. 1 is also a split closed 4-braid representative of K. 
We give it to show that the intersection of the splitting 2-sphere with the plane of 
projection need not be disjoint from the projected image of K. 


Similar phenomena arise when we consider composite links. A link K is 
composite if there is a 2-sphere Y in S* which meets the link in 2 points and 
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decomposes it into sublinks K, # K,, neither of which is an unknotted arc. A 
closed braid representative K of K is a composite closed braid if the 2-sphere Y can 
be chosen to meet the braid axis A in 2 points. An example is given in Fig. 2. One 
may ask whether every closed n-braid representative of a composite link is 
conjugate to a composite closed n-braid? That question has received some 
attention in the literature. Morton showed that every closed 3-braid which repre- 
sents a composite link is conjugate to an obviously composite 3-braid. In the same 
paper he also gave an example of a closed 5-braid which represents a composite 
link, but is not conjugate to a composite 5-braid. He was unable to decide whether 
4-braids resemble 3 or 5-braids in this regard. 

We will show in this paper how an arbitrary braid representative of a split (or 
composite) link can be changed to a split (or composite) braid representative. To 
state our contributions, we need the concept of an exchange move. In Fig. 3 the 
labels n; on the strands are weights, indicating n; parallel strands. We allow any 
type of braiding on the n, + n, (resp. n, + 3) strands in the boxes which are 
labeled X(resp. Y). Assume that the braid axis A is the z axis, and that the arc 
which is labeled n, lies in the x — y plane. Up to isotopy of S*, an exchange is 
defined to be an isotopy of K which moves the arc which is labeled n, from a 
position which is a little bit above (or below) the x — y plane to a position which is 
a little bit below (or above) the x — y plane, keeping the rest of K invariant. An 
exchange move takes n-braids to n-braids, but need not preserve conjugacy class 
because the isotopy of K in S? is in general not realizable in the complement of the 
axis A. The first two results in this paper show that exchange moves are the only 
obstruction to representing split (or composite) links by split (or composite) closed 
braids: 


The split braid theorem. Let K be a split link, and let K be an arbitrary closed n-braid 
representative of K. Then there exists a split n-braid K° which represents K and a 











Fig. 2. Composite closed braid representing the connected sum of two trefoils 
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finite sequence of closed n-braids: 


K =K,—-K, -K,-... ~K,, = K’ 


such that each K;, , is obtained from K; by either isotopy in the complement of the axis 
or an exchange. 


The composite braid theorem. Let K be a composite link, and let K be an arbitrary 
closed n-braid representative of K. Then there exists a composite n-braid K* which 
represents K and a finite sequence of closed n-braids: 


K = K,-K, ~K,- 


such that each K;, , is obtained from K; by either isotopy in the complement of the axis 
or an exchange. 


The theorem of Markov ([Ma] or [Bi] or [Be] or [Mo, 2]) shows that, since 
the 4-braids a, and a, of Fig. 1 (or Morton’s two 5-braids) represent the same 
oriented link type, they are related by a sequence of moves which involve first 
increasing and then decreasing the string index. This process is known as stabiliz- 
ation. Our theorems replace the stabilization process with a more direct process 
which changes one conjugacy class to another, preserving string index. It is thus a 
step in the proof of a version of “Markov’s theorem without stabilization”. A 
subsequent paper in this series will prove such a theorem in full generality. 

The braid index of a link K is the smallest integer n such that K can be 
represented by an n-braid. As an immediate corollary of the Split and Composite 
Braid Theorems we obtain: 


The braid index theorem. Braid index is additive (respectively additive minus 1) under 
disjoint union (respectively connected sum). 


Our next result relates to special versions of the Split and Composite Braid 
Theorems which hold when n = 3 and 4. 


The 3 and 4-braid theorem. (i) Every 3-braid representative of a split link is conjugate 
to a split 3-braid representative. However, there are examples of 4-braid representat- 
ives of split links which are not conjugate to split 4-braids. 
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(ii) Every 3-braid representative of a composite link is conjugate to a composite 3- 
braid representative. 


(iii) Let K be a 4-braid representative of a composite link K. If K cannot be 
represented by a closed braid of braid index < 4, then K is conjugate to a composite 
4-braid. However, if K can be represented by a closed braid of braid index < 4, thenK 
need not be conjugate to a composite 4-braid. 


2. Proof of the split braid theorem 


Our work begins with a split link K, and an arbitrary oriented closed braid 
representative K of K. The braid axis is A. We are also given a 2-sphere X which 
realizes the splitting of K. We orient K, and then orient A so that K is oriented in the 
positive sense about A. We assign (arbitrarily) an orientation to X, so that at each 
point of X there is a well-defined outward-drawn normal to X. The 2-sphere X will 
in general be pierced by the braid axis A many times. Our goal is to modify K until 
there is a 2-sphere X’ which realizes the splitting and is pierced twice by A. 

The principle tool in our work will be the study of the singular foliation of X 
which is induced by the fibration H of the open solid cylinder R* — A by half- 
planes {H,; te (0, 27]}. The leaves of this foliation are the components of X 7 H,, 
te€[0, 27]. The first step in the proof is to put X into a nice position relative to the 
fibration, in order to partially standardize the foliation. 

By standard general position arguments we may assume: 


(i) The intersections of A with X are finite in number and transverse. 


(ii) There is a solid torus neighborhood N(A) of A in S?-K such that each 
component of X q N(A) is a disc. 

(iii) The foliation in each component of X 7 N(A) is the standard radial foliation. 
(iv) All but finitely many H,’s meet X transversally, and those which do not (the 
singular fibers) are each tangent to X at exactly one point in the interior of H,. 
(v) The tangencies in (iv) are local maxima or minima or saddle points. 


A singular leaf in the foliation of X will be one which contains a point of tangency. 
All other leaves are non-singular. Note that it follows from (iv) and (v) that: 


(a) Each non-singular leaf is either an arc with both endpoints on A = GH, or a 
simple closed curve. 

(b) A singular fiber H, contains exactly one singular point pp. 

(c) Each singular point p, is either a center or a saddle. 


We now introduce a measure of the complexity of the pair (X, H). Let |X > A| be 
the number of points in X 4 A. Let |H- D| be the number of singular points in the 
foliation of X. The complexity c(X, H) is the pair (|X © Al, |H-D}). We assign the 
standard lexicographic ordering to this complexity function. We will say that 
(X, H) is equivalent to (X', H) if there is an isotopy taking X to X’ which takes 
(X OH,, X 0 0H,) to (X’OH,, X’ OV 0H,), for all te [0,22]. Notice that by our 
definition of equivalence, every representative of an equivalence class has the same 
complexity. 

We investigate simple closed curves (SCC’s) in X 7 H. We say that a pair (X, H) 
has SCC’s if there exists a non-singular fiber H, such that a component of X 1 H, is 
a SCC. 
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Lemma 1. Assume that (X, H) satisfies (i}{v) and has SCC’s. Then there exists a 
splitting 2-sphere X' such that (X*, H) also satisfies (i)-(v), and in addition has no 
SCC’s, moreover, c(X*, H) < c(X, H). 


Proof of Lemma 1. Our proof, sketched below, is very similar to the argument given 
in Lemma 2 of [B-M, 1]. The main difference is that our surface X is closed whereas 
the surface D in the proof of Lemma 2 of [B-M, 1] had non-empty boundary 
oD = K. 


If there is a SCC a(t) in X 0H, for some non-singular H,, we proceed as in the 
proof of Lemma 2 of [B-M, 1], following a(t) as it evolves in the fibration, until we 
arrive at a simple closed curve «(@) which contains a singularity of the foliation. 
Now «(@) lies on a singular fiber H,, and bounds a disc 4 in Hy. Notice that A 
cannot be punctured by K. For, the algebraic and geometric intersection numbers 
of K with A coincide, because A is a subdisc of H, and K is a closed braid with axis 
OH, . There is also a second disc A’, in X, with 0A = 0A’. Since K does not intersect 
X, it also does not intersect 4’. Now, 4U 4’ forms a 2-sphere S in S°, and the 
algebraic intersection number of S with K is zero because S is a 2-sphere. Since the 
geometric intersection number of A with K is equal to the algebraic intersection 
number of K with §, it follows that K does not pierce A. 

If the interior of 4 has empty intersection with X, we surger X along 4. The 
surgered surface X will bea pair of two 2-spheres, X (1) and X (2). At least one of 
these will be a new splitting 2-sphere, say X (1), and as in the proof of Lemma 2 of 
[B-M, 1] the new splitting 2-sphere will have complexity which is no more than 
that of the original X. If both X (1) and X (2) are splitting 2-spheres, it won’t matter 
which one we retain. If the interior of 4 meets X, we find an innermost subdisc 6 of 
A whose boundary is a component of X 7 H, and surger X along 6. Ultimately, we 
will arrive at a splitting 2-sphere X’ which has the property that the induced 
foliation has no simple closed curves, and c(X°*, H) < c(X, H). O 

By Lemma 1, each component of intersection of a non-singular H, with our 2- 
sphere X is an arc with both of its endpoints on A = OH,. Such an arc divides H, 
into two components, so it bounds two discs 4 and A’ on H,,. If either of these, say 
A, is not pierced by the link K, we can push our 2-sphere X inward along a 3-space 
neighborhood of 4 to remove two points of intersection of A with X. See Fig. 4. Of 
course, this could add simple closed curves to the foliation of X, but the definition 
of our complexity function was chosen so that even if it does, the complexity will be 
reduced, because we will have reduced | A 7 X|. Moreover, even if SCC’s are added, 
they can be removed with the aid of Lemma 1. 

This motivates us to call a leaf B in H,©X essential if K pierces both 
components of H, split along B. See Fig. 4 again. In view of the above remarks and 
Lemma 1, we can replace assumption (v) by the stronger assumptions: 


(v)* The tangencies in (iv) are saddle points. 
(vi) If H, is non-singular, then each component of X 1 H, is essential. 


We return to the main thread of the argument. The surface X is closed, so the braid 
axis A must pierce X an even number of times, say 2y. If 2u = 2, our closed braid is 
a split closed braid, and we are done, so assume 2 > 2. Thus each component of 
intersection of X with a non-singular H, is an arc which joins two of the 2 points 
where the braid axis A pierces the 2-sphere X. Recall that we assumed (see (iii) 
above) that the foliation of X is radial near each point of A q X. From this and the 
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fact that 2 > 2 it follows that there must be singularities in the foliation. Each 
singularity is the result of a saddle-point type tangency of a singular fiber H, with 
X. Figure 5 shows how the foliation looks in a neighborhood of a singular point. 
There will be four singular leaves which go out from the singular point like the 
spokes on a wheel, and end at points where the axis A pierces X. 

Let p;,..-, P2, be the points of An X. Let 0,,... , 0, be the t-values at which 
singularities occur. The singular leaves are then the leaves in {H,X; 
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6=6,,...,0,} which contain the singular points. Their complement in X can 
only be a union of regions R,,..., R,, as in Fig. 6, each of which is foliated 
without singularities. Choose a leaf e; from each R;. The union of all of these leaves, 
{e;,..., &} gives a cell decomposition of X. The 2y points py, ..., p2, where A 
pierces X are the 0-cells, the non-singular arcs {e,,..., e,} are the 1-cells, and the 
2-cells each contain one singularity of the foliation. Every 2-cell has four vertices 
and 4 edges (Fig. 5). We call our 2-cells tiles, and the foliated cell decomposition a 
tiling of X. 

The sign of a tile is defined as follows: each tile contains exactly one singularity 
of the foliation, which occurs at a point of tangency between X and a fiber H, of H. 
At the point of tangency the normals to X and H, coincide. The sign of a tile is 
positive or negative, according as the orientations on the normals to X and H, agree 
or disagree. Notice that if (@ — ¢,@ + €) is a t-interval about 6 which does not 
include any other singularities, then we can distinguish the two cases in the 
following way: Choose any t€(@ — ¢, @ + e) and label the sides of the arcs in H, 0 X 
“+” or “—” according as they correspond to the positive or negative sides of the 
oriented surface X. Then as t-@ at a positive (resp. negative) singularity the 
+ (resp. — ) sides of two arcs of H, 1 X will appear to coalesce, as in the left-to- 
right (resp. right-to-left) sequence in Fig. 7. 
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The valence of a vertex in the tiling is the number of tile edges which meet at 
that vertex. The next few lemmas concern the existence of vertices of low valence in 
the tiling. We will ultimately use these to recognize when our link admits a 
complexity-reducing exchange move. 


Lemma 2. The tiling of X always contains either a vertex of valence 2 or 3. 


Proof of Lemma 2. Let V, E and F denote the number of 0, 1 and 2-cells in the tiling. 
Then V— E+ F=2, the Euler characteristic of X. Now, each 2-cell in the tiling has 
four edges in its boundary, and each edge is an edge of exactly two 2-cells, so that 
2F = E,andso2V — E = 4. Nowlet V; be the number of vertices of valence i in the 
tiling. Since there are no vertices of valence 1, we have V= V,+V,+V,+ .... 
Since each edge has 2 vertices in its boundary, we also have that 
2E = 2V,+3V3+4V,+.... Thus 


2V,+V3=8+ V5+2V5+3V,+ ea ar 


The terms on both sides of this equation are non-negative. Thus there is always a 
vertex of valence 2 or 3. O 


Lemma 3. Either the tiling of X has a vertex of valence 2, or else there is an isotopy 
of X to a new splitting 2-sphere X’, such that the tiling of X' has a vertex of valence 2. 
Moreover c(X’, H) S c(X, H). 


Proof of Lemma 3. If the tiling of X already has a vertex of valence 2 there is 
nothing to prove. If not, then by Lemma 2 the tiling has a vertex p of valence 3. 
There are 3 tiles which meet at this vertex (see Fig. 8a) and two of them (call them T 















































Fig. 8. a Vertex p has valence 3; b flow TU 7’; ¢ proposed changes in tiling TU T’ 
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and 7’) necessarily have the same sign. Let their singular points be s and s’. Label 
the six vertices of TU T’ 1,2, .. . ,6 in cyclic order, so that the vertex p = 2, as in 
Fig. 8b. Assume the origin to have been chosen so that the singularity at s occurs 
before the singularity at s’. 

The future singularities have been indicated in Fig. 8b by two “joining arcs”, 
labeled «, and a,. We now explain what these mean. See Fig. 9, which shows the 
surface X and a fiber H, of H just before one of the saddle point tangencies. We see 
two arcs, « c X and f c H,. Together they bound a disc A in 3-space. These are our 
joining arcs. As H, is pushed up the arcs « and f will shrink, vanishing at the instant 
of tangency. Now, in our situation there will be two singularities (at s and s’), and so 
there will be two pairs of joining arcs: a, and «, on X (illustrated in Fig. 8b) and B, 
and f, on H, (illustrated in Fig. 10a). The idea of our proof is to show that we can 
deform X so that the two singularities are very close together both on X and in the 
fibration (we will make this precise below), and then interchange the order in which 
the singularities occur. The sequence of four pictures in Fig. 8c illustrate what we 
intend to do, from the point of view of an observer who is looking at the changing 
foliation of X during the deformation. After the deformation the tiling of TU T’ 
will have been changed so that the vertex p of valence 3 becomes a vertex of valence 
2, also the tiling will be unchanged in X — TU T’. We now show that these changes 
are always possible. 

The first thing we want to prove is that the six tile vertices 1, 2, . . . ,6, which are 
points on both X and A, must also have cyclic order 1, 2,.. . ,60n A, as illustrated 
in the Fig. 10a, which depicts a fiber of H before the two singularities. There is no 
natural choice of a normal bundle on X, so we assign one arbitrarily so that A 
(oriented either way) pierces X from the + side at the vertex 1. Note that this 
determines how A pierces X at every other vertex, because if the flow viewed from 
the + side of X is clockwise about any one vertex, then it is anticlockwise about 
any vertex which is joined to the given one by a leaf in the foliation. So, proceeding 
from vertex to vertex, the sense of the flow is determined everywhere. This, in turn, 
determines whether the oriented axis A pierces X from the + side or the — side at 
each vertex. 

Let ij(t) denote a leaf in the foliation which joins the vertices i and j. In the given 
tiling of X (Fig. 8b) the first singularity is between 12(t) and 56(t). This shows that 1 


singular point 
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and 2 cannot separate 5 and 6 on A = OH,. Thus the cyclic order of the four 
vertices 1, 2, 5, 6 on the oriented curve A is 1256, 2165, 1265 or 2156. However, the 
latter two are impossible because in the future surgery the sides of 12(t) and 56(t) 
which coalesce must have the same sign, and the sign is determined by our 
condition on the pierce-points. The first two choices are equivalent up to sym- 
metry, and determined by whether the surgery is to be “+” or “—”. We choose the 
order 1256, dictating that the first surgery is to be a “+” surgery. Similar 
considerations apply to the pairs 2, 5 and 3, 4 giving two possible cyclic orders for 
the 6 points: 123456 and 125436. However, with the latter choice the two singulari- 
ties will have opposite signs, so the order must be 123456 as illustrated in Fig.10a. 

We now have enough information to determine how Tu T’ is embedded in 3- 
space. Choose 6 cyclically ordered points on A and declare them to be the tile 
vertices 1,2, .. . ,6. Then choose two fibers of H and declare them to be the fibers 
H,, and H,, which are to contain the singularities s and s’ respectively. Then 
choose points s and s’ in the fibers and declare them to be s and s’. Up toa 
homeomorphism of 3-space and a reparametrization of the fibers these choices are 
all arbitrary. Finally, join up s to the points 1, 2, 5, 6 by arcs in Hy, which have that 
cyclic order when H,, is viewed from the + side, and similarly join s’ to the points 
2, 3, 4, 5 by arcs in H,,, using that cyclic order when Hy, is viewed from the + side. 
Thus we have embedded the singular leaves through s and s’ in 3-space. The next 
thing to do is to adjoin little discs to the points 1, 2, . . . ,6. These discs are to be the 
intersections of X with a neighborhood of A in 3-space. Since A pierces X from the 
+ side (resp. — side) at 1, 3, 5 (resp. 2, 4, 6), we know how to embed these discs as 
oriented surfaces. There is then a unique way to extend the embedding to a 
neighborhood of the singular leaves in 3-space. Finally, let 1 be an arc which joins s 
to s’ in X, and let N be a foliated neighborhood of y, as in Fig. 8c. Then there is a 
unique way to adjoin N to the part of TU T’ already constructed. Finally, extend 
the embedding to all of TU 7’. The result is depicted in Fig. 11a. (We have removed 
a tubular neighborhood of the axis A from 3-space, so the points 1, 2,... , 6 are 
replaced by circles, seen here as vertical arcs.) For comparison we have also shown 
the embedding when the signs disagree, in Fig. 11b. 

As noted earlier, we want to push s and s’ together on X and in the fibration and 
then to interchange their order in the fibration. There is clearly no obstruction to 
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Fig. 11. a Non-degenerate saddles at s and s’ signs at s and s’ agree; b non-degenerate saddles at s 
and s’ signs at s and s’ do not agree 


pushing s and s’ close together on X. We now investigate whether there are 
obstructions in fibers of H. We first ask whether parallel sheets of X could interfere 
with the project (as they clearly might if the signs at s and s’ disagree). Inspecting 
Fig. 10a, one potential difficulty becomes clear. It is possible that there is another 
leaf in the foliation of X, e.g. the leaf 3°4°(t) illustrated in Fig. 10b, which is parallel 
to the leaf 34(t) and obstructs the change we wish to make. It is not hard to see that, 
since we know that the singularity between 12(t) and 56(t) is possible, this is the 
only obstruction which can occur. After the first singularity the picture will be as in 
Fig. 10c. Now, since by hypothesis the singularity between 25(t) and 34(t) is 
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possible, there must in fact be a pair of leaves which are parallel to 34(t) in Fig. 10b, 
i.e. 3°4°(t) and 3" 4"(t), and a singularity (indicated by the joining arc y) between 
them which occurs before the singularity between 25(t) and 34(t). Now notice that 
on the surface X the four tile vertices 3°, 3”, 4°, 4” are disjoint from T and T’ (and so 
not visible in Fig. 11a). Thus there is no obstruction to changing the fibration so 
that the singularity between 3°4°(t) and 3" 4"(t) occurs before that between 12(t) 
and 56(t), removing the obstruction. 

Since K intersects every fiber of H, we must also show that K cannot obstruct 
our proposed change in the order of the singularities. (Notice that K could be an 
obstruction in the situation of Fig. 11b, because if we were to push s down we 
would force K to be tangent to a fiber of H, so that it would no longer be a closed 
braid). Refer now to the sequence of pictures in Fig. 12. The top one is a repeat of 
Fig. 10a, with a little new data added. Initially, K will pierce one of the four regions 
of H, split along the three arcs of H, 7 X. By symmetry it suffices to consider three 
such choices. We have indicated these by a square black dot, a horizontal slash and 
an X in the three regions. We follow the time-evolution of the dot. It can wander 
anywhere within a region as t varies, but it cannot cross from one region to another 
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because K does not intersect X. After each singularity, new regions open up to it. 
We have shown the final position of the square black dot, if the left sequence is 
used. We now see that we could have achieved the same end position if, instead, the 
right route had been used. The same argument applies to the slash and to the X. 
For every possible position it is possible to proceed either via the left or instead via 
the right sequence. Thus K is not an obstruction to the change in order of the 
singularities. 

The proof is essentially complete, however we can improve our understanding 
of the changes by showing precisely how to do the deformation of Nc TU T’. We 
may regard N as the graph of the smooth function f(x, y) = y? — 3y(x? — 1). This 
function has two non-degenerate saddle-type critical points of the same sign, one at 
(x, y) = (1, 0) and the other at (x, y) = (— 1, 0). Now, our function f(x, y) belongs 
to a parametrized family of functions f,, with f(x, y) = y? — 3y(x? — e?) and f=f,. 
If ¢ + 0, the function f£(x, y) has a pair of critical points at (+ ¢, 0). Both critical 
points are saddles because at (+ ¢,0) the Hessian is — 36e” < 0. Deform X to a 
new surface X’ by letting the parameter ¢ pass to zero. This has the effect of pushing 
s and s’ toward one-another, until at ¢ = 0 the two saddles coalesce to a monkey 
saddle, which has a single degenerate critical point at (0, 0). The graph X’ of f(x, y) 
has three “hills” and three “valleys”. The singular leaves (i.e. the level sets fo '(0)) 
are three lines in the x — y plane through (0, 0), of slope 0, ,/3 and — ,/3. The 
surface X’ has local 3-fold rotational symmetry about the origin. We can then 
further deform X° by rotating it through, say — x/,/3, allowing ¢ to increase again, 
and then rotating it back again, i.e. replace f, by g~' fg, where g is the rotation. 
This has the effect of splitting the critical point into two new critical points along 
the line of slope ,/3. 0 


Lemma 4. If the tiling contains a vertex of valence 2, then K admits an exchange. 
Moreover, after the exchange the complexity can be reduced by removing two or more 
of the points where the axis pierces X. 


Proof of Lemma 4. There are two tiles which meet at our vertex of valence 2, and the 
first step in the proof is to examine the signs of the singularities in these two tiles. 
Let p be the vertex in the statement of the lemma, and let q and s be the singular 
points of the two tiles which meet at p. Let p’ and p” be the other vertices of the two 
tile edges which meet at p. The singular leaves through q and s occur at t, and fz, 
respectively. Portions of these singular leaves fit together to cut off a disc 6 on X, 
and the region depicted in Fig. 13 is a neighborhood A of 6 on X, chosen so that 04 
is everywhere transverse to the foliation. We have labeled leaves of the foliation as 
occurring at times t = f,,f2,..., tg. 

We now examine corresponding leaves as they would occur on a sequence of 
fibers of H. See the sequence of five pictures in Fig. 14, which are labeled 
t3, t4, ts, te, t; to correspond to the labels in Fig. 13. The full cycle is obtained from 
this one by adding to it the corresponding sequence of pictures run backwards. The 
singularities which occur at t, and t, have opposite signs. (This is a direct 
consequence of the fact that there are exactly two singular leaves among the leaves 
which are incident at p). Thus the normal bundles to X and H agree (say) at s, and 
disagree at q. This is the first fact which we need, in order to see how A is embedded 
in 3-space. 

The next thing to notice is that (from Fig. 5) the gradient flow on X is always 
oriented in opposite senses around the two endpoints of‘each leaf which joins two 
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points of Am X. This implies that 04 has algebraic rotation number | about the 
axis A. Since 0A is everywhere transverse to the foliation, it then follows that 0A is a 
1-braid, and also that the axis A pierces A from alternating sides at the 3 pierce- 
points, say the positive side, the negative side, and then the positive side again at 
p, p’ and p”. 

We now have enough information to determine how A is embedded in 3-space. 
See Fig. 15. The first step is to choose three points on the axis A and declare them 
to be p’, p and p”. Up to a homeomorphism of S? which fixes A and each fiber it will 
not matter where we place them. Next, we attach small discs transverse to the axis 
with their centers at p’, p and p”. These discs are the intersections of our oriented 
surface X with a neighborhood of the axis, and by the argument we have just given, 
if the disc at p’ has its positive side up, then the discs at p and p” will have negative 
and positive sides up, respectively. 

The next step is to choose points on the fibers at t, and t, and declare them to 
be q and s. Up to a homeomorphism of S* which preserves A and each fiber it will 
not matter where we place them. Now, each singular point is the intersection of 
four singular leaves of the foliation, which go out from the singular point like the 
spokes on a wheel. Three of the leaves which meet at q and also at s terminate at 
p’, p, p’, SO we can extend them (in their fibers) to p’, p and p”. The fact that the 
singularities have opposite signs shows that if the cyclic order (looking down onto 
the positive side of X) at q is p’, p, p”, then it must be p”, p, p’ at s. The fourth leaf at s 
and also at s’ goes to the boundary of 4, which (as noted earlier is a little 1-braid 
about A. Since the surface A is transverse to the foliation in the complement of the 
singular leaves, there is now a unique way to complete the embedding of A. In this 
way we obtain Fig. 15. 

We study the non-singular leaf b(t,) in Figs. 12 and 13. By hypothesis, every leaf 
in the foliation is essential. This means that K itself is necessarily an obstruction to 
their removal, as illustrated in Fig. 15. We propose to remove this obstruction by 
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an exchange, as illustrated in Fig. 16. After the exchange we will obtain a new tiling 
with the same complexity, however in the new tiling the leaf b(t,) will be inessential. 

The remaining problem is to show that we can actually realize this exchange so 
that it takes braids to braids and leaves the tiling unchanged, except for reordering 
of the p;’s. If we can do this, then our exchange move will enable us to reduce the 
complexity. We need to create a region in 3-space in which we can make the 
exchange move in a controlled fashion. 

Each non-singular H, meets X in a unique arc b(t) which joins p to p’ or p” and 
cobounds with a part of the axis A two discs in the fiber H,. If b(t) joins p to p’ (resp. 
p”), let yu, be the disc which does not contain p” (resp. p’). We have sketched in one 
such arc b(t) and shaded in the disc y, in Fig. 17. If we think of the subsurface 4 of 
Fig. 15 as a boxing glove, the discs py, for t,; < t < ts will appear to be “outside” the 
glove, whereas those for t; < t < t, will be “inside” the glove. The disc 6, which is 
on the glove, is a limiting position for both families of discs. Thus the closure of the 
union of all of the discs y,, t€ [0, 27] will be two 3-balls B, and B,, which intersect 
along a single arc in the disc 6 of Fig. 13, i.e. the arc which runs from q to s through 
p. The shaded disc in Fig. 14 is in the boundary of B, U B. 
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We may assume that B, U B, intersects the link K in two unbraided weighted 
arcs. This is clearly possible, since all braiding may be pushed out of B, UB. 
Notice that the surface X may meet B, and B, in some number of sheets which are 
locally parallel to the embedded disc 4 which we depicted in Fig. 15. In particular, 
X may meet a neighborhood Ny, of A in some number of radially foliated discs 
between the discs at p’ and p”, as in Fig. 18. 

Now we need to thicken B, UB, a little bit. With this in mind, reparametrize 
the interval [0, 27] so that the singularities occur at t, = 0 and t, = 2. We can then 
pair the discs y, and y,,,, So that py, is in B, and p,,, is in B,. See Fig. 19, which 
depicts subsets of H, U H,..,,. Now notice (see Fig. 13) that if H,, 0 < t < 7, is non- 
singular, then H, contains two leaves in the foliation of 4: the leaf b(t), which joins p 
to p”, and also a small arc a(t) which runs out from p” to the boundary of A. 
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Similary, p,,,, is a union of b(t + 2), which joins p to p’, and a small arc a(t + 2) 
which runs out to the boundary from p”. See Fig. 19. Let N(t) be a neighborhood of 
HV yee UO(t) Ua(t + 2)in H,UH,,,, chosen so that N(t) OK = (u,U 4,4,) OK 
and N(t) q X is the union of (u, U »,,,) 0 X and (H, U H,,,,) 0 Ng. Finally, choose 
the neighborhoods N(t) so that they vary smoothly as t is varied between 0 and z. 
Let N be the closure of the union of all of the N(t), t¢[0, 2). Notice that N is not a 
3-ball (it is a 3-ball with holes), because N(t) is not a disc for t-values close to t = 0. 
Chosse ¢ > 0 so that N(t) is a disc for te[e, x — é]. 

We can now describe precisely the exchange move which was indicated earlier 
in Fig. 16. See the top left picture in Fig. 20, which is intended to correspond to the 
left picture in Fig. 19, at some te(e, z — ¢). The plane of the paper is divided by A 
into two half-planes, and we will assume the left one to be H, and the right one to be 
H, , ,. We want to describe an isotopic deformation of X U K, and shall do so by the 
series of pictures in Fig. 20, all of which correspond to the same fixed value of 
te[e, x —e]. 

Choose a subdisc d of y,,, such that dn K =(y,,,) 0K and also dn X 

= (u,4,) 0X — b(t + 2). Our isotopy is to be supported in N(t) and is to be the 

identity on p, and on p,,, minus a neighborhood of d on p,,,. The isotopy pushes 
the disc d (and the points of K and X which meet it) across A and then down and 
eventually into the cross hatched area below p” which was illustrated in the blow- 
up in Fig. 19. The isotopy is to be defined on pairs of fibers, and is to be defined so 
that it varies continuously as we vary t. At the end of the isotopy the link K is to 
encircle the axis below p” instead of between p’ and p. Also, each sheet of X which 
intersected A between p’ and p is to intersect A the same number of times below p”. 
(One of the authors likes to think of this change as accomplished by “putting your 
hand into your pocket and emptying it”.) 

The only remaining problem is to ask what happens in the t-interval [0, ¢]? Let 
u be the isotopy parameter. The first thing to notice that when u is in the interval 
[.25, .75] the isotopy is supported in the left half-plane. We may then assume that 
the deformed discs d,, ue [.25,.75], have empty intersection with H,UH,,, if 
t¢ [e, 2 — ¢]. The second thing to notice is that when ue [0, .25] and we [.75, 1] the 
isotopy is supported in a neighborhood of the axis A. Since that neighborhood can 
be chosen to be disjoint from a neighborhood of the singular points (see Fig. 14) it 
follows that Fig. 20 actually tell us everything we need to describe the isotopy of 
X UK completely. 

At the end of the isotopy we may reposition X so that the tiling of X will be 
exactly as it was before the change. The only change will be in the order of the 
points of A X on A. The interchange of order has an important consequence: one 
or more essential b-arcs in the foliation of X will have been changed to inessential 
b-arcs. We can therefore modify X by an additional isotopy to a new splitting 
2-sphere X’, reducing the complexity. The proof of Lemma 4 is complete. 0 

We now complete the proof of the split braid theorem. We begin with a splitting 
2-sphere X which has complexity c(X, H) 2 (2, 0). If the complexity is (2, 0) we are 
done, so assume it is > (2,0). This implies that |A 7 X| > 2, because if it were 2, 
then by our general position hypotheses we would also have |H:X|=0. By 
Lemmas 2 and 3, we can then conclude that there is a vertex of valence 2 in the 
foliation of X. By Lemma 4 we conclude that after an exchange we may replace X 
by a new splitting 2-sphere X’ with smaller complexity. After finitely many such 
changes we will obtain a splitting 2-sphere X” which intersects A twice, and a split 
n-braid representative K’ of our link K. 0 
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3. Proof of the composite braid theorem 


We begin with a closed n-braid K which represents a composite link K. By the split 
braid theorem, we know that if K is a split link, we may find splitting 2-spheres 
which exhibit it as an obviously split closed braid. Thus we may work on one 
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component at a time. Therefore we may assume that our representative K is non- 
split. 

The proof is almost identical with that of the split braid theorem, therefore we 
will only discuss the points where they differ. We are given a 2-sphere Y (replacing 
X) which decomposes K into a connected sum of non-trivial link types. The only 
difference between the 2-sphere X of Sect. 2 and our 2-sphere Y is that X was 
disjoint from K, whereas K pierces Y twice. Our goal is to modify K and Y until we 
achieve a 2-sphere Y’ which also realizes the connected sum decomposition and 
which is pierced twice by the axis A. 

The general position arguments given in Sect. 2 go through, but a complication 
occurs in the proof of Lemma 1, when we try to remove SCC’s from the foliation of 
Y. The possibility exists that K will pierce the discs which are to be used in the 
surgeries. Suppose that c is a SCC in H, 4 Y. Then c bounds a disc 6 in H,, and also 
divides Y into two discs 6, and 6,. Since K intersects Y twice, it either intersects 6, 
once and 6, once, or (by choosing the notation appropriately) it intersects 6, twice 
and misses 6,. Assume the latter. Then if we surger Y along 6, as in the proof of 
lemma 2, Y will be split into two 2-spheres Y, = 6U6, and Y, = 6 U6, and Y, 
must be inessential because it misses K. Thus we can replace Y by Y,, and proceed 
as in the proof of Lemma 2. If, on the other hand, K meets both 6, and 6,, then 
both 2-spheres Y, and Y, will intersect K twice, and at least one of them, say Y,, 
must be essential. We can then discard Y,, and proceed as in the proof of Lemma 2, 
Thus we may assume that Y is foliated without SCC’s. 

Our 2-sphere Y admits a tiling. Call a tile good if it is not pierced by K. Call a 
vertex of valence 2 or 3 good if it is adjacent to a good tile. As in the proof of the 
split braid theorem, we have to prove there is a vertex of valence 2 or 3 in the tiling, 
but now we need a little more: we must be sure that the vertex of valence 2 or 3 is 
good. The equation. 


2V,+V3=8+V5,+2V,+3%,+4+ eeee 


still holds. In this equality every entry is non-negative. Now, we can assume (after a 
small isotopy of K) that the two points where K pierces Y are in the complement of 
the set of singular leaves, i.e. in a region R like the one depicted in Fig. 6. Call R a 
bad region if it is pierced by K. Now, a region R is adjacent to exactly two tile 
vertices, and since there are at most 2 bad regions this means that there are at most 
4 bad vertices. Thus, if V, + V3, 2 5, we can always find a good vertex, so assume 
V,+V3<4. This means that V, < 4. However, V, + (V, + V3) 2 8. Therefore 
the only case where we could fail to have a good vertex of valence 2 or 3 is in the 
very special situation when 2V, = 8 and V, = 0. This means that Y is a union of 
exactly two tiles, also the complement of the set of singular leaves is a union of four 
regions, and two of these are pierced by K. 

We can now proceed, as in the proof of Lemma 4 above, to construct Y as an 
embedded surface in S*. Let p,, p2, P3, P4 be the 4 points where A pierces Y, in their 
natural cyclic order on A. Each of these vertices has valence 2, so there are 2 tiles 
and 2 singular points in the foliation, say s and q. See the top pictures in Fig. 21. 
Since all of our vertices are bad, we know that K must pierce a pair of opposite 
regions, say R, and R3. By the first part of the proof of Lemma 4, we know that the 
signs of s and q are opposite, so the clockwise cyclic order of the tile vertices will be 
Pi» P2» P3 Pa in, say, the tile which contains s and p,, p3, Pz, P; in the tile which 
contains g, when viewed by an upright observer on the positive side of the tile. The 
points s and q lie in distinct fibers, and the singular leaves will be a union of four 





136 J.S. Birman and W.W. Menasco 


arcs which go out from s(resp. q), in a single fiber, like the spokes in a wheel, ending 
at the four cyclically ordered vertices, on A in a neighborhood U of the singular 
leaves the 2-sphere Y will be transverse to each fiber, so there is a unique way to 
extend the embedding of S to an embedding of U. With U in place, we can attach 
the four non-singular regions R,, R,, R3, R,. The embedding of Y is then as in the 
bottom picture of Fig. 21. 

By hypothesis each arc in the foliation of Y is essential, so our link K must be an 
obstruction to its removal. See Fig. 22. Thus K must encircle the axis n = 1 times 
between p, and p, and also m 2 | times between p; and p,, inside Y. These strands 
will in general be braided; we have indicated the braiding as occurring inside two 
boxes, labeled A and B. Similarly, K must encircle the axis j = 1 times between p, 
and p; and k 2 1 times between p, and p,, outside Y, with braiding occurring in the 
boxes which are labeled C and D. In addition, one strand of K pierces Y in the 
region R, and another in the region R;, to join up the part of K which is inside Y 
with the part which is outside Y, as illustrated. Thus K is a connected sum of a link 
which is represented as an (n + m)-braid and a link which is represented by a 
(j + k)-braid. 

It is now clear that we can slide the strands of K which pierce Y into, say, 
Region 2. After such a slide we will have a good vertex of valence 2. A complexity- 
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reducing exchange move is applicable. The proof of the composite braid theorem is 
complete. 0 


4. Proof of the 3 and 4-braid theorem 


In this section we prove special versions of the split and composite braid theorems 
which hold when the braid index is 3 or 4. See Sect. 1 above for the statement of the 
theorems. 

Two of the four cases have been settled elsewhere, vis: In [Mu] Murasugi 
proved, in effect, that every closed 3-braid representative of a split link is conjugate 
to a split 3-braid representative. The analogous situation for composite links was 
settled by Morton in [Mo, 1]. A third case, that of split braids of braid index 4, has 
not been treated elsewhere, however the examples which we gave earlier in Fig. 1 
show that there are split links of braid index 4 which are not conjugate to split 
closed braids. Thus the only case which requires proof is the case of composite 
braids of braid index 4. 


Remark. The reader will have no difficulty in supplying new proofs for the three 
other cases, with the techniques which we will use to settle the fourth. 
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We begin, as in the proof of the composite braid theorem, with a composite link 
and a 2-sphere Y which realizes the connected sum decomposition. We assume, as 
usual, that Y is in a nice position relative to H, so that Y has a tiling. Each tile has 4 
vertices and 4 edges and 1 singularity in its foliation. The first step will be to 
examine the components of H, 7 Y as they occur in a sequence of non-singular 
fibers at times t = a, b,c,.... See Fig. 23. Let a be a component of H, 7 Y. Then « 
separates H,, and since « is essential (see Sect. 2) both components of H, split along 
a must be pierced by K. On the other hand K is a closed 4-braid, so we know that K 
pierces H, in exactly 4 points. This implies that « is parallel to one of the 4 arcs 
types illustrated in Fig. 23. 

We assume that there are n,, nj, 3,4 parallel arcs of each type, where each 
n; = 0. If we follow any one arc as we sweep through the fibration, we know it must 
ultimately be modified by a surgery. However, arcs in the position of B and 6 in 
Fig. 24 cannot be surgered with one-another because such a surgery would create 
an inessential arc. Similarly, arcs like « and » cannot be surgered with one-another, 
because that too would create an inessential arc. Thus the only possibility is that 
arcs in the position of « and t are surgered with one-another. This has the effect of 
removing « and t from a pair of opposite groups and adding new arcs «’ and 1’ to 
the other pair. This process must continue until there are no more arcs left in one of 
the pairs, at which point the process must be reversed, because every arc must both 
be modified by a surgery and also ultimately go back to its initial position. 
However, this means that eventually «’ will be surgered again with 1’. Thus, in fact, 
if there were more than two arcs our surface Y would be disconnected. Hence there 
are exactly two arcs in H, 4 Y, and two singular fibers. 

The tiling of Y thus contains exactly two tiles, each with four vertices of valence 
two. This is precisely the situation which was described in the proof of the 
Composite Braid Theorem in Sect. 3, so we can conclude that the picture is exactly 
that in Fig. 22, with n = m = j = k = 1. Note that this implies immediately that our 
link K is actually the connected sum of three, not just two links, each of which can 
then only be a type (2, r;) torus link, for some r; = 1. Thus we have proved that if K 
is the connected sum of a link of braid index 3 and a link of braid index 2, then K is 
in fact conjugate to a composite 4-braid. 

We go back to the case where K has three factors, and construct the 2-sphere 
which realizes the second connected sum decomposition of K = K, # K, # K3. 
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Referring momentarily to Fig. 21, let c, be an arc on Y which joins the points s and 
q in the interior of R,, and let c, be an arc which joins them in the interior of R3. 
Then c = c, Uc; is a simple closed curve which divides Y into two discs, a disc Y, 
which contains R, and a disc Y, which contains R,. (Thus p, and p, are in Y,, 
while p, and p, are in Y,). We assume that we chose c, and c, so that the two 
points where K pierces Y are in Y,. Then Y, is a disc with boundary c which is 
disjoint from K, and is pierced twice by A, at p, and at p,. Keeping the curve c in 
mind, we now pass to Fig. 22, where we see that there is another disc, Z which also 
has boundary c, which is chosen so that the interior of Z lies “outside” the 3-ball 
bounded by Y. In fact, we can choose Z so that it intersects K in two points, along 
the strands which join the blocks labeled C and D, and so that Z 7 A is empty. Let 
Y’' = Y, UZ. Then Y’ is also a 2-sphere which realizes the connected sum de- 
composition of K, and Y’q A = 2 points. Thus, except for the very special case 
when the block D consist of a single crossing, we see that K is an obviously 
composite closed braid. 

In the case where D contains a single crossing, our closed braid K has a trivial 
loop. That is, K is a connected sum of type (2, r, ) and type (2, r,) torus links, r; = 1, 
and it is represented as a closed 4-braid. In this case there actually may be a non- 
composite 4-braid which represents K. The proof is complete. 0 


References 


[Be] Bennequin, D.: Entrelacements et equations de Pfaff. Asterisque 107-108, 87—161 
(1983) 

[Bi] Birman, J.S.: Braids, links and mapping class groups. Ann. Math. Stud. 82 (1974) 

[B-M, I] Birman, J.S., Menasco, W.W.: Studying links via closed braids I: A finiteness 
theorem. Preprint 

[B-M,II] Birman, J.S., Menasco, W.W.: Studying links via closed braids II: On a theorem of 
Bennequin, Topology and its Applications (to appear) 

[B-M, III] Birman, J.S., Menasco, W.W.: Studying links via closed braids III: Classifying links 
which are closed 3-braids. Preprint 

[B-M,V] _ Birman, J.S., Menasco, W.W.: Studying links via closed braids V: The unlink. Trans. 
AMS. 

[G] Garside, F.: The braid groups and other groups. Q. J. Math. Oxford 20 (No. 78), 
235-254 

[Ma] Markov, A.A.: Uber die freie aquivalenz der geschlossenen Zopfe. Rec. Soc. Math. 
Moscou 43, 73-78 (1936) 

[Mo, 1] Morton, H.R.: Closed braids which are not prime knots. Math. Proc. Camb. Philos. 
Soc. 86, 421-426 (1979) 

[Mo, 2] Morton, H.R.: Threading knot diagrams. Math. Proc. Camb. Philos. Soc. 99, 
247-260 (1986) 

[Mo, 3] Morton, H.R.: Infinitely many fibered knots with the same Alexander polynomial. 
Topology 17, 101-104 (1978) 

[Mu] Murasugi, K.: On closed 3-braids. Mem. AMS 151 (1976) 











Invent. math. 102, 141-156 (1990) perc 
entiones 


mathematicae 
© Springer-Verlag 1990 





Effectiveness—non effectiveness in semialgebraic 
and PL geometry 


F. Acquistapace, R. Benedetti, and F. Broglia* 


Dipartimento di Matematica, Universita di Pisa, Via F. Buonarroti 2, I-56 127 Pisa, Italy 


Oblatum 14-IV-1989 & 25-I-1990 


0. Introduction and statement of the main results 


Working in real semialgebraic geometry one could easily have the feeling that the 
main facts can be realized by algorithms (e.g. path connecting points, triangul- 
ations, good stratifications . . . ) and the main trouble is to construct reasonably 
fast algorithms: in fact it is this (in principle) constructive nature of semialgebraic 
geometry and the existence of powerful computers which make it interesting in 
view of concrete applications. 

On the other hand, Nabutovsky’s examples of non recursive functions associ- 
ated to natural semialgebraic constructions (see [N]) tell us that there are some 
conceptual limitations to the above attitude. 

In this paper we consider some further examples of effectiveness-non effect- 
iveness results in semialgebraic and PL geometry: our starting point was the study 
of Shiota-Yokoi solution of the subanalytic (hence semialgebraic) Hauptvermutung 
({[SY]) and the attempt to make it effective. 

Before starting we need to introduce some notations. 

We shall use systematically the notation: 


[A;,.-., 4a) +8... 


to mean that there exists an algorithm accepting the “objects” A,,..., A, as input 
and producing B,,..., B, as output. 

With the same meaning we shall say also that “B,,..., B, can be effectively 
constructed, starting from A,,..., A,’- 

Similarly we shall write: 


[A,,..-,A,]+>0B,,..., By] 


to mean that “B,,...,B, cannot be effectively constructed starting from 
A,,..-,A,”, that is there doesn’t exist any algorithm producing B,,... , B, from 
A, ee | A,.- 





* The authors are members of GNSAGA of CNR. This work is partially supported by MPI. 
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Some time we shall consider also systems: 


oe ..,4,) > [B,,..., By] 
iC: f -»€,)-(D,,. ‘ satel 
with the obvious meaning. 


A semialgebraic set is any subset X ¢ R" (some n) with a presentation of the 
form: 


ki 
she ij +0 


if 
where each f,,e R[X,,...,X,], *ij€ ity > }. 
For any such presentation set: 


r= Yk; d=sup{degree of f;} . 


By the notation X €.¢A(n, p, q), n, p, qe N, we mean that X is a semialgebraic set 
in R" with a given presentation such that r < p,d < q. 

A semialgebraic map f: X > Y between semialgebraic sets is a continuous map 
with semialgebraic graph I;,. 

If K is a simplicial complex in R" (some n), P =|K\| is the polyhedron in R" 
triangulated by K; #K denotes the number of simplexes in K. A PL m-ball is any 
polyhedron B < R" PL isomorphic to the standard m-simplex A,,. By K , we denote 
the standard triangulation of A,,. 

Let us state now some of the results of the paper. 


Theorem 1 (strong effective semialgebraic Hauptvermutung for dimension < 3; 
shortly: SEH,, m < 3). Let K, L be finite simplicial complexes in R". #K, #L < k. 
Let f:|K|—|L| be a semialgebraic homeomorphism with I, € oJ (n’, p,q). Assume 
m = dim |K| = dim |L]| < 3. Then 
at L,fJ>[g:K'>L’] 
[n, k, p,q] > [D] 

where K’ is a simplicial subdivision of K, L’ of L, g is a simplicial isomorphism and 
#K’', #L’ < DEN (see Theorem 3.5). 


Theorem 2 (weak effective semialgebraic Hauptvermutung for any dimension; 
shortly: WEH,, Vm). With the same notation as in Theorem 1 but with arbitrary 
m = dim|K| = dim|L], one has 


[K,L,f]>[g:K'>L’] 
(see 3.3). 

In the PL setting we have the following effective trivialization of PL ball: 
Theorem 3. Let B =|K| be a triangulated PL m-ball, |K| <¢ R", #K <d,m< 3. 
Then 

ae >[g:K’'>L] 
[d] > [D] 


where K’ is a simplicial subdivision of K, L is a subdivision of the standard 
triangulation of A,,, g is a simplicial isomorphism, De N and # K' < D (see Problem 
2.2 and Proposition 2.23 and 2.24). 
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Remark. If m < 2 we are able to omit the hypothesis |K| © R”, that is we may 
assume |K| ¢ R", n + m in general. 


Theorem 4. With the same notation as in Theorem 3, if m is arbitrary and | K| < R’, 
with m + n in general, then 
[K]—>[g: K’>L] 


(see Problem 2.1 and Corollary 2.17). 


Remark. Theorems 3 and 4 are, in some sense, PL counterparts of Theorems | and 
2; in fact they shall be employed to prove the first ones. 

The SEH,, for m 2 4 is still open (to us). However the “PL counterpart” is false, 
that is we have the following non effectiveness result for trivialization of PL-balls of 
higher dimensions. 


Theorem 5. (a) For every m,n,deN there exists s = s(m,n,d)eEN such that for 
every PL m-ball B =|K\| & R’, triangulated by K with #K <d, there exists a 
simplicial isomorphism g: K'— L where K’ is a subdivision of K,L of K,, and 
#K’' Ss. 


(b) For every m= 6 
[m,d]+[D] 


such that DEN and s(m, d) = s(m, m, d) < D (i.e. s(m, d) cannot be bounded by any 
recursive function of m, d) (see Corollary 2.18 and Remark 2.21). 


In the paper we are concerned with the opposition ‘effectiveness-non effect- 
iveness”; we shall not consider the problem of the complexity of the algorithms 
(when they exist). 

We shall assume the existential solution of the (semialgebraic) Hauptvermu- 
tung by Shiota- Yokoi; however the eventual effectiveness (strong or weak) is not a 
consequence of their proof, not even in low dimensions: we shall discuss shortly the 
reason of it in (3.10). 

For basic facts about semialgebraic geometry we refer to [BCR] or to [BR] (in 
the second one some effectiveness questions are explicitly pointed out). 

For basic facts about PL-geometry we refer to [RS] and [Hu]. 


1. Recall on a result of Novikov 


Integrating Smale’s h-cobordism theorem (and its consequences) with the results of 
Adyan, Markov, on the “non-decidability of the triviality for finitely presented 
groups”, Novikov (see [VKF]) proved the following result (in a formulation which 
is more convenient to our aim). 


Theorem (Novikov). For every m = 6 one can construct a sequence M = {Ms = 
IKsl}s en Of triangulated PL submanifolds of R", dim Ms = m, 0Ms + &@ such that 


(i) [|Ksle 4] +[t] 
where teé { yes, no} and t is “yes” if Ms is a PL m-ball, t is “no” otherwise. 


(ii) (|Ks|e 4] >[t] 
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where t€ { yes, no} and t is “yes” if OMs is a PL (m — 1)-sphere, i.e. PL isomorphic to 
OA»), “no” otherwise. 


Novikov’s theorem shall be one of the main tools in order to prove Theorem 5 
of the Introduction. 

The idea to use it (in fact a smooth analogous of it) in order to produce 
examples of non recursive functions in semialgebraic geometry is due to Nabutov- 
sky (see [N]). 


2. On the effective trivialization of PL-balls 


We shall adopt the notation of [RS], in particular: we distinguish between m-cells 
and PL m-balls; if A, B are cells A < B means that A is a face of B; if K and L are 
(cell or simplicial) complexes, L<i K means that L is a subdivision of K; St(A, K) 
denotes the star of A in K. 

By definition a PL m-ball is PL isomorphic to the standard m-simplex A,,. We 
denote by K, the standard triangulation of A = 4,,. 

By a trivialization of B we mean a simplicial isomorphism g: K — L where 
B=|K| and LaK,. 


Problem 2.1 (WT,,,: weak effective trivialization problem in dim ™m). It asks for 
[K]—>[g: K’>L] 


where B =|K| is any triangulated PL m-ball, K'1K, L<1K,, g is a simplicial 
isomorphism. 
Problem 2.2 (ST,,: strong effective trivialization problem in dim m). It asks for 
ee >[g: K’'>L] 
[d] > [D] 


where B = |K| is any triangulated PL m-ball, with #K < d, K’'aK, La K,,g isa 
simplicial isomorphism De N and #K’ S D. 


We shall see, in particular, that WT,, has solution for every m; on the contrary 
ST,,, has not solution for m = 5. In low dimension we have some stronger results. 


(A) Recall of some simple facts of PL-geometry 


We shall use systematically the following well-known facts which we state with 
emphasis on effectiveness. 


Lemma 2.3. For any cell-complex K with a number of cells #K S d one has 
[K]>[H] 
[d] > [D] 


where H is a simplicial complex such that Ha K, #H < DEN, the 0-skeleton H® and 
K° coincide. 


Proof. See [RS], Proposition 2.9. 0 
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Lemma 2.4. If K, M are two simplicial complexes such that |K|=|M|, #K, 
#M < d then one has: 
[K,M]>[H] 
[d] > [D] 
where H is a simplicial complex such that Hai K, M, #H < DEN. 
Proof. Apply (2.3) to the cell complex Ka M. O 


Two cell-complexes K, L are said formally isomorphic if there exists a bijection 
q: K > L such that Vo, te K o S t> (oe) S q(t). 


Lemma 2.5. If K, L are cell-complexes, q: K > L is a formal isomorphism, # K, 
#L <d then one has: 


ao 
[d] > [D] 
where H, M are simplicial complexes, H<1 K, M< L, f is a simplicial isomorphism, 


#H = #M SDeN. 


Proof. The construction of H in Lemma 2.3 uses a suitable order on the cells of K. 
Induce the order on L via qg. Construct M as in 2.3 using this order. One gets two 
formally isomorphic simplicial complexes, hence simplicially isomorphic. 0 


(B) Effective shelling and effective trivialization of PL balls 
Let B =|K| be a triangulated PL m-ball. 


Definition 2.6. A shelling of K is an ordering of the m-simplexes of K: 


ee 
such that for everyi=1,...,h—1 
B;:= B\ |) oa; (By = B) 
1Sjsi 
is a PL m-ball (triangulated by the restriction of K). 
Remark 2.7. If m < 2 every triangulation K of a PL m-ball B carries some shelling 
(for m = | is trivial, for m = 2 see [Mo]). 

On the contrary, if m = 3 there are triangulations of PL m-balls (in fact of A,,) 
without any shelling (see [R]). It is known (see [BM; S], and also [Ac]) that for any 
triangulation of any PL m-ball there exists a subdivision admitting some shelling. 

The proof (see [BM]) in the general case is not effective and uses the existence 


of a PL isomorphism with 4,,. In fact we want to prove that the effectiveness of 
shellings (up to subdivision) is equivalent to the effectiveness of trivializations. 


In analogy with the trivialization problems we can state: 
Problem 2.8 (WS,,: weak effective shelling problem in dim m). It asks for 
[K]—>[K’, 7] 
where B = |K| is any triangulated PL m-ball, K'=1K, and ¥ is a shelling of K’. 
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Problem 2.9 (SS,,: strong effective shelling problem in dim m). It asks for 
ms > [K’, F] 
[d] > [D] 
where B = |K|, K'<1K, ¥ are as above, #K <d, DEN and #K’' SD. 
We want to prove: 


Proposition 2.10. For every m, the strong (weak) trivialization problem for PL m- 
balls has solutions if and only if the strong (weak) shelling problem has solutions. 
Shortly: WT,, <> WS,, and ST,, <> SSm- 


Proof. We shall prove ST,,,<>SS,,. The proof works also in the weak case. 
(a) ST,, => SS,, 


It is enough to find solutions of SS,, for any KK, with #K <d. This is a 
particular case of a more general statement. 


Lemma 2.11. If C = |K| is any triangulated m-cell in R” (i.e. C is a convex m-ball), 
#K <d then one has 


[K]>[K’, 7] 
[d] > [D] 
where K'aK, #K’' < DEN, and ¥ is a shelling of K’. 
We postpone the proof. 
(b) ST,, = SS,, 
Assume, for the moment, the following lemma: 


Lemma 2.12 (effective trivialization of collars of 04,,). If L<1 K,, #L < p then one 
has 


te + [L',h:C > M] 
[p] > [4] 


where L'a L, #€L' < qeEN, |M| = 0A,, x [0,1], C is a subcomplex of L’ triangu- 
lating a collar of 0A,, and h:C—+M, is an effective trivialization such that 


We shall prove Lemma 2.12 later. It is enough to find solutions of ST,, for any 
triangulation K of B=|K|, with #K <d and a given shelling o,, o,...,4¢,, 
where r is the number of the m-simplexes of K. We do it by induction on r. For 
r = | it is trivial. Assume that we have done for (r — 1) and try to do for r. Set 
|K,| = K \o,: it is (by hypothesis) a PL m-ball triangulated by the restriction of K, 
say K,. # restricts to a shelling , = (o2,...,0,) and K, has (r — 1) m-simplexes; 
#K, <d. 

By induction we have a solution of ST,, for K,, that is we construct effectively 
K, <1 K,, L,;<1K,, 9,: K > L, a simplicial isomorphism, D, € N (depending on 
d) such that #K’, <D,. 

By pulling-back via g, a collar of 04,, in 4,, given by Lemma 2.12 we may 
assume also that there exists a subcomplex T of K’, such that |7| is an effectively 
trivialized collar of 6|K’,| in |K’;|, by h: T > M where |M| = 0|K‘| x [0, 1), hyaix-: 
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0|K',| > 0| K{| x {0} is id x {0}. We want to construct a simplicial isomorphism 
f: | on K{ 

where K< K, Ki <1 K’, and #K is effectively controlled by d. Thus we shall 

complete the proof by taking the simplicial map g = g, °f:|K|—> 4,,. 

o, intersect |K,| along 0|K,| and the union of the (m — 1)-faces of K, in 
0, 0 0|K,| is an (m — 1)-ball B’ = o, 7 6|K,|. Using the trivialized collar we have 
that B” = h~'(B’ x [0,1]) is isomorphic to B’ x [0, 1] with an effectively given 
simplicial isomorphism (for a suitable subdivision). To conclude it is enough to 
show that id: 0B"\B’ > 0B’\B' extends to a PL isomorphism q:o,|_),-B’ > B” 
which becomes simplicial for suitable subdivisions effectively constructed with 
bounds on the number of simplexes. 

Set B’” = o,|_)g-B’. First remark that the complement in 0B” of the interior of 
0B’ \B’ coincides with the complement in 00, of the interior of B’ and also that 
0B” = B’U OB’ \B’. 

Both B’ and dc,\B' are cones over their intersection B’ 7 éa,\B’, hence we can 
extend q to an isomorphism q’: 0B” > 0B’. 

Finally, using again the effective trivialization of the collar, we can realize both 
B’” and B” as cones over their boundary; hence we can extend to q”: B’’ > B’. 
Remark that this is the proof of 3.25 of [RS], which becomes effective because of 


the effective trivialization of the collar. The proposition is proved (assuming 
Lemma 2.12). UO 





Remark 2.13. The same proof shows also that ST,, and SS,, are equivalent to the 
following 


[d]>[D] 


where B = |K|, is any triangulated PL m-ball, |K’| = B and is a shelling of K’; 
#K <d, #K’' < DEN (that is we do not require that K’< K). Similarly for the 
weak statements. 


ie >([K’, #] 


Proof of Lemma 2.11. With a slightly different statement the proof is contained in 
[Ac]. For the sake of completeness we sketch the construction with emphasis on 
effectiveness. The proof is by induction on n. 

Let B be a convex triangulated n-polyhedron, B = |K|. We shall describe an 
explicit subdivision which is shellable. 

Consider a family E,,...,E; of parallel hyperplanes with the following 
properties: 


i) EgnK = {v°}; v° is a vertex. 
ii) each vertex v of K is in some E,. 
iii) In a suitable coordinate system 


E,;={x,=4;} with a <a,<...<ds 


It is easy to see that one can do this effectively. 

E, 0 B is a convex (n — 1)-polyhedron (or possibly a single vertex). By Lemma 
2.4 and the induction hypothesis we can subdivide explicitly the cellularization 
induced by K on E,-. B in a shellable triangulation K;. Let 


F, = Bo {a <x, £4;} Ss 


i-i = 





148 F. Acquistapace et al 


F, is a convex polyhedron with a cellularization induced by K, K;_, and K;. We 
can find a triangulation of F;,i = 1,. . . , s without introducing new vertices. So we 
obtain a linear simplicial subdivision K’ of K such that the vertices of K’ lie in 
E,u...UEs. Define: 


V,, = {o€ K'|\dimo =m,oc F,,dim(o VN E;_,;)=k-—1}. 


If ce V,, and o’ c F; is such that o and o’ have a common (m — 1)-face, then 
ao EV, .-,U Vi, 0 V;,.441- SO we can define the worst type simplices in V;,, as the 
simplices which have no common (m — 1)-face with any simplex in V;, ,4,. 

It is rather easy to prove, as in [Ac], that the following rules define a shelling Y 
for K’. 


(1) decreasing i. 
(2) increasing k. 
(3) for fixed i and k, first the worst type simplices with more free faces. 


So the lemma is proved. 0 


Proof of Lemma 2.12. By means of Lemma 2.4 it is enough to construct an 
effectively trivialized collar of 0A,, for some triangulation of A4,,. Consider A,, as the 
cone over 0A,, with center in the barycenter a of A,,,. 

Every veA,, is uniquely expressed as v = ta + (1 — t)v’ where te[0, 1] and 
v’ €0A,,. Set: 


A(s) = {veA,,|v =sa+(1—s)v' for some v'edd4,,} OSs<1 


Note that 4(0) = 04,,. Consider A(4) ¢ A,,. Using the cone structure we find a 
natural cell-decomposition of A, having as vertices the union of the vertices of 4(0) 


and A(3) and such that the induced cell-decomposition on |_)o<;< ,4(s) is formally 
isomorphic to the standard cellular decomposition of 0A,, x I (i.e. the product 
K 4\0,, * 1). Thus we can conclude using Lemma 2.4 and Lemma 2.5. 0 


Remarks 2.14. (i) The proof of 2.13 works for any m-cell in R". 

(ii) Consider K = K,, and let K” denote the rth-barycentric subdivision of K. One 
can explicitly construct (it is elementary but not completely trivial) a simplicial 
isomorphism between the subcomplex of K), St(@A,,, K) with a suitable sub- 
division of 0A,, x I, (obtaining in this way another proof of 2.12): PL topology tells 
us (see [RS]) that if M=|K| is a triangulated manifold with boundary, 
St(@M, K) is a collar of 6M in M. The problem to us is to make the trivialization 
effective. We have not been able to do it in general, not even under the hypothesis 
that M is a PL ball (but K is arbitrary); in fact there are some evidence that it 
should not be possible in higher dimensions see (Remark 2.19 and Corollary 2.20). 


(C) Listing bounded subdivisions and effective trivialization of PL balls 

The aim of this section is to prove the following: 

Proposition 2.15 (listing bounded subdivisions). Let X = |K| be any triangulated 
polyhedron in R" with # K < d and fix a positive integer D = d then we have 


ee - IK|J >C{H,, a) H,}] 
[d, n, D] > [S] 


where SEN and s SS, for every i=1,...,s, H;\a K, #H; < D, and for every 
H=< K such that # H < D there exists i and a simplicial isomorphism $: H > H;. 
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Before proving the proposition we deduce some corollaries. In particular we 
prove Theorem 4 and Theorem 5 of the Introduction. 


Lemma 2.16. Given X =|K|, Y =|H| triangulated polyhedra, we can effectively 
decide if there exists a simplicial isomorphism f. K — H. 


Proof. In the worse case (that is if there are not evident distinctions like 
dim X + dim Y or # K + # H) we have to look for formal isomorphism between 
K and H, thus we have to analyze a finite number of possibilities. O 


Corollary 2.17 (solution of the weak trivialization and shelling problems). For every 
m WT,, and WS,, have solution. 


Proof. Since WT,,,<> WS,, (Proposition 2.10) it is enough to show that WT,, (for 
example) has solutions. Take a triangulated m-ball B = |K|. By hypothesis there 
exists some K"<1 K, #K’=R,, L<1K, such that g: K’>L is a simplicial 
isomorphism. If r= # K, for every R 2 r we can list (Proposition 2.15) all the 
subdivisions of K with no more than R simplexes and all the analogous sub- 
divisions of K ,. Using Lemma 2.16, for every R we can check if there are any such 
subdivisions which are isomorphic. By hypothesis there exists some Ry such that 
the algorithm stops. 0 


Corollary 2.18 (non effectiveness for the strong trivialization and shelling prob- 
lems). For m 2 5 ST,, and SS,, do not have solution. More precisely: ifm = 6 they do 
not have solution even under the stronger hypothesis that the m-balls are embedded in 
R”; if m = 5, ST, and SS, fail even for codimension one 5-balls in R°®. 


Proof. Assume first m = 6 and ST,,, has a solution. Take any triangulated manifold 


M; =|Ks| S R" given by Novikov theorem recalled in Sect. 1. If # Ks < d and if 
Ms should be a PL m-ball one could effectively determine De N such that there 
exists a simplicial isomorphism g:K,—~ LL, where K;<1 Ks and #Kj,<D, 
L,; <1 K,. Using the listing of bounded subdivisions as in the previous proof one 
could decide if M, is isomorphic to 4,, (by actually finding isomorphic subdivisions 
with no more than D simplexes or concluding that they are not PL isomorphic 
otherwise). This contradicts Novikov’s theorem. 

Let be m = 5 and suppose that ST, has a solution. Take again M, = |Ks| ¢ R®, 
take off a simplex o from the boundary 0M, and consider N; = 0Ms\c. Ng is a 
triangulated 5-manifold and by ST, and Lemma 2.16 we can decide whether N, is a 
5-ball or not. So we can decide whether 0M, is a 5-sphere or not and finally, by h- 
cobordism in dimension 5, we can decide whether M, is a 6-ball or not, contradic- 
ting Novikov’s theorem. 0 


Remark 2.19. As a consequence of Corollary 2.18 we cannot have a strongly 
effective proof of h-cobordism theorem: indeed, if so, taking off a simplex t from the 


interior of a triangulated m-ball B, one could explicitly triangulate B\t as the 
product 0B x I; this would be enough to have ST,,. A rough analysis of the proof of 
h-cobordism theorem shows that there are two a priori non effective steps: the first 
is the trivialization of the handle decomposition, where trivialization of collars is an 
essential tool; the second is the choice of a disk bounded by the curve of “Whitney 
trick”. 

Actually 2.18 has also a rather surprising corollary: collars cannot be strongly 
effectively trivialized. 

We thank A. Marin who remarked it. The argument is the following. 
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Recall that, if not specified otherwise, by PL m-balls we mean balls generally 
embedded in some R", n and m not necessarily equal. Given a PL m-ball B with 
boundary S, a trivialized collar of S in B is a PL homeomorphism between a regular 
neighbourhood N of S in B and S x [0, 1], identifying identically S and S x {0}. 
We have, with the obvious meaning, a strong effectiveness problem for construction 
of such trivialized collars. In case of positive answer, we say shortly that collars can 
be trivialized. 


Corollary 2.20 (non effectiveness for the trivialization of the collars). There exists m, 
3 < m <7, such that collars in m-balls cannot be trivialized. 


Proof. Let m, the minimal number m such that ST,,, (extended to all balls) has 
negative answer. By Corollary 2.18 mo exists and in fact 2 < mp < 6. Then one 
deduces that (m, — 1)-spheres can be strongly effectively trivialized on the bound- 
ary of the (m,) — 1)-simplex (otherwise we contradict the minimality of mg). For 
every m,-ball B consider the (my + 1)-ball B’ obtained by coning over B. If collars 
of the boundary S’ of B’ can be strongly effectively trivialized, the above remark on 
the trivialization of the boundary of B together with the unicity of links (which is a 
strongly effective result) imply that B can be strongly effectively trivialized. This is a 
contradiction. 0 


Remark 2.21. Since Novikov’s manifolds Ms are of codimension 0 in R™, we have 
proved the point (b) of Theorem 5 stated in the Introduction. It remains to prove 
the point (a). It is a consequence of a more general fact: 


Lemma 2.22. For every n,déN there exists D = h(n, d) such that for every couple of 
triangulated polyhedra X =|K\|, Y =|H| SR" with # K, #H <d, if X, Y are PL 
isomorphic then there exist subdivisions K'<1 K, H'< _ H with # K', # H’ < D which 
are simplicially isomorphic. 


Proof. It is a consequence of (the proof of) the theorem of local triviality of 
semialgebraic maps as stated and proved, for example, in [BCR] Theorem 9.3.1, 
following the proof of Hardt ([Ha]) (see also [BR], 2.8.3.). Note that in our case the 
situation is much simpler because we are working with PL objects. 

Every triangulated polyhedron in R" with no more than d simplexes can be 
presented using no more than d (effectively computed by means of n and d) 
polynomial of degree 1, is such a way that it is given as the union of its simplexes. 

We can identify the polynomial in n variables of degree 1 with R"*' and set for 
every h<d 

R= R, sie (R"*1)* 
Set: 
S=#S,={>,=,<} 


Take R" x R with the natural projections 
tT R"x ROR q: R" x R> R" 
For every t = (t,,..., t,)e€S consider 
V, = {(x, P,,..., P,)eR” x R: P;(x)t,0} 
Finally set for every k: 
V=Vi~={VIVHV,,U...0V, teeS 1Sisk} 
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¥ isa finite set. It is easy to find an effective upper bound B for k (in term of d) such 
that every triangulated polyhedron with no more than d simplexes is of the form 
X =q(nx~'(v°) AV) for some v°ER, VeY, h,k < B. Apply the local triviality 
theorem to the maps 7)»: V +R, V varying in ¥. It turns out from the proof that 
Ty is trivialized over a finite number of semialgebraic subsets of R giving a 
partition of R (in fact in our case they are defined by polynomial of degree 1). More 
over we note that: 


(1) For every V the number of such subsets of R is effectively bounded in term of n 
and d (hence of d). 

(2) #V is effectively bounded. 

(3) Two fibers of the same trivialization are PL isomorphic by a simplicial 
isomorphism between subdivisions with bounded number of simplexes (see [BR] 
where all this facts are explicitly noted). 


Eventually we may have fibers of different trivializations which are nevertheless PL 
isomorphic. Anyway there is a finite number of possibilities hence the function 
D = h(n, d) exists. O 


Proof of Proposition 2.15. It is again an application of the proof of the local 
triviality for semialgebraic maps ([Ha; BCR; BR ]) adapted (and simplified) in our 
setting. Given a subdivision K'< K, X =|K|, # K’ < D then the polyhedron 
X =|K| & R" is presented as union of the simplexes of K’ using no more than D 
polynomials of degree 1, and D is effectively computable using n, d and D. Let 
P,,..., P, be such polynomials. They produce a cell-decomposition of X by cells 
of the form: 


6, = {xEo|P;(x)t,0} 


where o is a simplex of K, t=(t,,...,¢,) and t; is a value in {>,=,<} 
associated to P,, i=1,...,7r. 

It is clear that such a cell -decomposition subdivides K and has a number of cells 
effectively bounded. 

On the other hand every subdivision K’ < K with # K’ < Dcan be obtained by 
some cell decomposition of the previous type (that is defined by no more than D 
polynomials of degree 1) by deleting some cells: since the number of the cells is 
effectively bounded, then there is only a finite number of possibilities to delete cells 
in order to get (if any) a required subdivision of K. 

We work similarly to the proof of Lemma 2.22. 

Let A ¢ R"*' be the subset of polynomials of degree 1 with zero set intersecting 
X. Set for h < D: 


R =R, = A’ S=S,={>,=,<}° 
mR" x ROR gq: R" x R- R" 
For every t = (t,,..., t,)ES set 
V, = {(x, Pj, ..., P,)EX x R: P,(x)t,0} 
V=V,,=(VIV=V,,U...0V, teS 1Sigk} 
k < B effectively bounded 


Apply the local trivialization theorem to my: V +R, V varying in ¥. For every 
réR my’ (r) is a cell-subdivision of X of the type described above. 
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It turns out from the proof of the triviality theorem that, in particular, two 
fibers in the same trivialization of x over some R; of the partition of R are formally 
isomorphic; hence the subdivision of K obtained by deleting faces with the same 
name in both such fibers are simplicially isomorphic. Since the number of R; is 
effectively bounded the proposition is proved. Note that in this way we may obtain 
several representatives of simplicial isomorphism class of subdivisions of K pre- 
sented by no more than D polynomials; what is important is that we get at least one 
representative for each class. 0 


(D) Strong effective trivialization of PL balls in low dimension 


In this section we shall prove, in particular, Theorem 3 of the Introduction. 


Proposition 2.23. SS,, and ST,,, have solution for m < 2. 


Proof. Since SS,,<>ST,, (2.11) it is enough to show SS,, (for example). Now as 
already noted in 2.8 every triangulation of a PL m-ball with m < 2 carries some 
shelling and at least one of these can be detected effectively. O 


Proposition 2.24. SS, and ST have solutions for PL 3-balls embedded in R°. 


Proof. ST is a consequence of an effective version of the proof of the PL Sh6nflies 
theorem in R*; we follow Moise’s statement (see Theorem 17.12 in [Mo]), whose 
proof is very similar to the original one (see [A1]). Since what follows Lemma 1, p. 
122 of [Mo] is straightforwardly effective, it is enough to prove the effectiveness of 
the first part. Here are the steps. 


(1) The effective push property EPP 


Let C be a polyhedral 3-cell in R? and D, < 6C* bea polyhedral 2-cell. Let J be the 


boundary of D, and define D, = 0C\D,. We say that C has the EPP at D, if for 
each closed neighborhood N of C\ J one has: 


ig H'-L] 
[d] > [D] 
where K, G, H are compatible linear triangulations of C,D,, N, #K, #G, #H < d; 


H’= H; L is a linear triangulation of N, #H’', #L<D and f is a simplicial 
isomorphism such that 


(i) Sion = id 

(ii) f(D,) = D, 

C has the EPP if this is true at any polyhedral 2-cell D < 6C. 
(2) Any convex polyhedral 3-cell has the EPP 


This is true since proofs of Theorems 4-9 of Chap. 17 in [Mo] are (or can be easily 
reduced to) effective proofs. 


(3) Simply embedded spheres 


We say that a polyhedral sphere S is simply embedded if it bounds a polyhedral 3- 
cell B and ST; is true for B. 
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(4) Theorem 17.11 in [Mo] 


Theorem. Let S,, 5, be polyhedral 2-spheres in R? such that S, 0 S, is a plane 2-cell 
D. Let 


S =(S, US,)\int D 


If S, and S, are simply embedded, then so also is S. 
Proof. See Theorem 17.11 in [Mo], using EPP instead of the push-property. 
(5) The index of a polyhedral 2-sphere 


Define the index of a polyhedral 2-sphere as in [Mo]: first choose a system # of 
orthogonal coordinates x, y, z in R* in such a way that each plane z = const. does 
not contain more than one vertex of S: this can be done effectively. For any 
horizontal plane E, ES can be: a single point, a (disconnected) 1-manifold, the 
disjoint union of a 1-manifold and a single point, the union of k polygons, 2 or more 
through a point P. In the latter case we call P a singular point and define the index 
at P, IndgP, to be k — 1. Then define IndgS, as the sum of the numbers Ind gP for 
all the singular points P in S. 


(6) Lemma. If S is a polyhedral sphere and IndgS = n, then one can effectively find a 
coordinate system &' and two polyhedral spheres S,, S, such that S is simplicially 
isomorphic to their union (as in the step 4) and Indg, S; <n—1i=1, 2. 


Proof. (see Lemma 17.1 of [Mo], p. 122). Let P be a singular point of the section 
ES, where E = {z = zg} in the system &, and let JG ES be the inmost 
polygon. J bounds a 2-cell D, in E. 

The only remark to be made is that the PL homeomorphism h: R? > R? 
constructed in the proof of [Mo], which preserves horizontal planes and makes D, 
convex, is simplicial on an effectively computable triangulation of S and R?: but 
this is clear because of SS, and Lemma 2.5. 0 


Remark 2.25. SS,, and ST,,, for m = 4, are open (to us), even under the restriction to 
4-balls in R*. 


The above facts suggest the following: 
Problem 2.26 (embedding problem (E,,)). It asks for 


te > [g: K'> H] 
[d] > [D] 


where B = |K| is a triangulated PL m-ball, # K < d; K'1 K, DEN and # K’ SD, 
g is a simplicial isomorphism, |H| is a PL m-ball embedded in R”. 


3. On the effectiveness of the solutions of the real semialgebraic 
Hauptvermutung 


We recall some well-known facts in semialgebraic geometry (see [BCR] or [BR]). 
We use the notations introduced in Sect. 0. 
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Theorem 3.1 (semialgebraic triangulation). Let X €.W(n,s, D) be a compact semi- 
algebraic set in R". Let Y;€.o(n, s, D) be a closed subset of X,i=1,...,h. Then 


i Y,]>[K,f] 
[n, s,D] > [r, p,q, R] 


where |K| is a triangulated polyhedron of R", f:|K|— X is a semialgebraic home- 
omorphism such that: f~'(Y;) =|K;| and K; is a subcomplex of K,i=1,...,h; 
T',€.A(p, q, R), #K Sr. 


We say that any f: |K| > X as above (without the limitation |K| ¢ R") is a 
semialgebraic triangulation of X relative to { Y;}. 


The following is an almost immediate corollary of 3.1. 


Corollary 3.2. Let f,: |K,| —~ X |= 1, 2 be a semialgebraic triangulation of X 
(relative to {Y;}). Let # K,< d, ,,E A(n,s, D). Then 


[hf] =" [K, h,, hz] 
[n, s,d, D] > [r, m, q, R] 


where for |= 1,2 h,: |K|—|K,| is a semialgebraic triangulation of |K,| relative to 
the family of all simplexes of K,, I,,€ ¢(m, q, R), #K <r. 


Thus we have a diagram 


and (|K|,h,,h,) is called a simultaneous semialgebraic subdivision of the trian- 
gulations f,, f, of X. Clearly (|K|, ffoh,) 1 = 1, 2 is a semialgebraic triangulation 
of X. 


(A) WEH , 


3.3. Proof of the WEH,, Ym (Theorem 2 of the Introduction). Let f: |K|—|L| be a 
semialgebraic homeomorphism between two polyhedra in R" as in the statement of 
the theorem. Using Corollary 3.2 we may assume that f satisfies the further 
hypothesis: for every simplex o of K, f(c) is triangulated by a subcomplex of L. 

By Shiota-Yokoi existential solution of the Hauptvermutung we know that 
B = f(o) is a triangulated PL m-ball, m = dima. 

By means of Theorem 4 of the Introduction (see 2.17) we can effectively 
construct a trivialization of B. Thus we conclude (as in [SY], p. 737) by induction 
on dim | K|, constructing effectively a PL isomorphism on the skeletons in increas- 
ing dimension and applying several times the “Alexander trick”. 0 


Remark 3.4. The same proof as above (in the same way of [SY], p. 737) produces in 
fact a more precise result: that is we can construct effectively a semialgebraic isotopy 
H:|K\| x [0,1] |Z] such that Hyp =f and H, =g, that is the simplicial iso- 
morphism of the statement of WEH,,. 


Theorem 3.5 (Theorem 1 of the Introduction). SEH, is true for m < 3. 
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Proof. For m S 2 we can work as in the Proof 3.3 using Proposition 2.23 instead of 
2.17. 

For m = 3 we want to reduce the Proposition 2.24, that is to work with 3-balls 
embedded in R°. 

Consider a semialgebraic homeomorphism f:|K|—|L| such that for every 
simplex o€ K, B = f(o) is triangulated by a subcomplex L, of L (like in the proof 
3.3). 

Take a 3-simplex o of K, clearly we may consider o € R°. 

g =f~': B=|L,|-¢ is a semialgebraic triangulation of o. Apply the semi- 
algebraic triangulation theorem and its proof (see for instance [BCR] or [BR]) too 
relatively to the family given by the images of the simplexes of L,. We obtain a 
straight triangulation of o, say o = |H| such that every simplex of L, goes by g’ 
over the support of a subcomplex of H, where g’: B > a is a suitable semialgebraic 
homeomorphism of effectively bounded complexity. 

Thus we can effectively construct a PL isomorphism between |L,| and o 
working like in the Proof 3.3 (induction, construction on skeletons of increasing 
dimension, Alexander trick) applying 2.25 to each g’(t) S o, t varying among the 3- 
simplexes of L,. 0 


Remark 3.6. (in analogy with Remark 3.4.). For m <3 we can construct the 
semialgebraic isotopy H, in such a way that [,€.~%(k, p,q) where k, p, q, are 
effectively bounded by the data of the situation. 


(B) SEH,, and semialgebraic homeomorphism of bounded complexity. 


At the present we are not able to decide if SEH,, holds for m 2 4. 
In this section we show simply that SEH,, and an other natural semialgebraic 


problem cannot be solved effectively at the same time. The spirit of the result is 
analogous to Nabutovsky’s examples. 


Lemma 3.7. For every m,n, dé N there exist P,QeN such that P,Q depend on n,m, d 
and for every X, Ye. (m, n, d) which are semialgebraically homeomorphic, there 
exists a semialgebraic homeomorphism h: X > Y with ,€ (m7, P, Q). 


Proof. It is essentially the same of 2.22, using the local triviality theorem of 
semialgebraic maps. 


Problem 3.8. Can P, Q as above be bounded by a recursive function of n, m, d? 


Proposition 3.9. For m = 5, the statement: “SEH,, has solution => Problem 3.8 has 
negative answer” is true. 


Proof. Let M = |K| be any compact polyhedron. If 3.8 has a positive solution we 
can effectively determine P, Q, depending only on m and # K, such that if M is 
homeomorphic with 4,,, then there exists a semialgebraic homeomorphism 
h: M > A,, with I’, €.o/ (m7, P, Q). If also SEH,, has-solution we can deduce that one 
can effectively determine D depending only on # K and m such that if M is PL 
isomorphic with 4,, then there exists a simplicial isomorphism g: K’ > L where 
K'< K, La K, and #K’ SD. This contradicts 2.18. 0 


Remark 3.10. The hard part in Shiota-Yokoi solution of subanalytic Hauptvermu- 
tung is in showing that if a polyhedron P = |K| is subanalytically homeomorphic 
to A,,, then it is PL isomorphic to A,, (see also the Proof of 3.3 of the present paper). 
Shiota-Yokoi proof doesn’t get any essential simplification in the semialgebraic 
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setting and it is far from giving SEH,,, not even WEH,,. The main reason of it is 
that Shiota-Yokoi use Cairns-Whithehead C“-triangulations and the (already 
known) solution of the Hauptvermutung for this class of triangulations (see [Mu]). 
This produce at least two troubles of increasing difficulty: 


(a) It is not clear if a semialgebraic manifold M € oA (n, r, p) could be triangulated in 
the C®-sense, M = |K| say, in such a way that # K is effectively controlled by 
n,r,q. The usual way (secant approximation ... (see [Mu])) doesn’t work: 
consider for example a sequence of manifolds M,é.W(n, r, q) converging to 
some cusps as in the figure. The semialgebraic triangulation algorithm (again 
for the cusp presence) doesn’t produce C ®-triangulations. 


(b) (More serious) Shiota-Yokoi apply the uniqueness of C®-triangulations to the 
diffeomorphisms obtained by integrating vector fields. Even if these vector fields 
are defined by semialgebraic functions of bounded complexity, how to bound, 
or even how to define, the complexity of the resulting diffeomorphism is a hard 
problem. 
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I. Introduction 


Let (R, m) be an excellent normal local domain of dimension two. Suppose that R/m 
is algebraically closed. In this paper it is shown that R has a rational singularity if 
and only if J? is a complete ideal for any m-primary complete ideal I of R. We give 
an example to show that this theorem is false if R/m is not algebraically closed. 

Related problems have been considered in [Z1; ZS; MS; L; G; HS]. This paper 
is a continuation of my study of complete ideals in [C; C1] and [C2]. 

Section III gives an example of a normal but not Cohen-Macaulay scheme 
which birationally dominates a regular local ring of dimension three. 

My main results are the following two theorems. 


Theorem 1. Suppose that (R, m) is an excellent and normal’ local domain of dimen- 
sion 2. Suppose that k = R/m is algebraically closed. Then the following are equi- 
valent. 


(1) R has a rational singularity. 

(2) If 1 and J are any two complete ideals in R, then the product IJ is a complete ideal. 
(3) If I and J are any two complete m-primary ideals in R, then the product IJ is 
a complete ideal. 

(4) If I is any m-primary complete ideal in R, then I? is a complete ideal. 


(/) implies (2) is proved by Lipman in [L]. 
However, this theorem fails to be true if k is not algebraically closed, as is shown 
in Theorem 2. 


Theorem 2. Let k be a field of characteristic not equal to 3. Let R(k) be the ring 
R(k) = kL Lx, y, z]]/x? + 3y? + 92°. 


* Partially supported by NSF 


1 The referee has pointed out that analytically normal is enough. The proof given here also works 
in this case, since we can replace R by its completion 
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The following is true 


(1) R(k) is anormal local domain of dimension two and a simple elliptic singularity. In 
particular, it is not rational. 

(2) Let k = Q be the rational numbers. If I, J < R(Q) are any two complete ideals, 
then IJ is a complete ideal. 

(3) If k has positive characteristic, or if k is an algebraically closed field, then there 
exists an m-primary complete ideal I < R(k) such that I? is not a complete ideal. 


Theorem 1 is proved in Sect. I and Theorem 2 is proved in Sect. II. 
Section III is devoted to giving an example showing interesting behavior in 
dimension three. 


Example. There exists a normal but not Cohen-Macaulay scheme Z obtained by 
blowing up an m-primary ideal J in R=C[[x,y,z]] with natural map 
h:Z — spec(R) such that R*h,0, = 0 and R*h,O, + 0. @,./” is normal but 
not S;. 3 


For clarification, we will give an indication of the main idea of the proof of 4) 
implies 1) in Theorem 1. 

Suppose that f:X — spec(R) is a resolution of singularities, J c R is an m- 
primary complete ideal, and IO, = ©,(D) is a Cartier divisor. Then I? is a complete 
ideal if and only if the natural map H°(0,(D))®? + H°(@,(2D)) is a surjection. 

Suppose that R is such that it has the particularly simple form that there exists 
a resolution of singularities f: X — spec(R) such that the reduced exceptional locus 
of fis irreducible. Call the reduced exceptional locus C. It is well known that R has 
a rational singularity if and only if C is a nonsingular rational curve. 

Assume that R satisfies the assumptions of 4) of Theorem 1, but that R does not 
have a rational singularity. We will derive a contradiction. 

By Lemma 5 below, there exists a nonsingular closed point P on C and B > 0 
such that if Y = O,.(BP), then Y is generated by global sections, H'(Y)= 
H'(Y®*)=0, and H°(#)®? + H°(Y®) is not surjective. There exist r > 0 
and nonsingular points Q,,...,Q, of C such that 0,(—rC)@ ©,(— BP) =~ 
O-(Q, +...+Q,). For simplicity, we will suppose that the Q; are all distinct. 

Let g:Y—X be the blow up of Q,,...,Q, with exceptional divisors 
F,,..., F,.Set D = g*(—rC) — F, —...—F,. Let C’ = C be the strict transform 
of C by g. By construction, 0,(D) ®@ Oc: = L. Oy(D) is generated by global sections, 
and H°(0,(D))—> H°(Oc(D)), H°(@y(2D)) + H°(Oc(2D)) are — surjections. 
H°(0,(D))®* + H°(O,(2D)) cannot be a surjection since H°(#)®? > H°(#®”) is 
not a surjection. J = H°(@,(D)) is an m-primary complete ideal since D has 
exceptional support. Hence I? + I? = H°(@,(2D)), a contradiction. 

I thank Bill Heinzer for kindling my interest in the problem addressed in this 
paper. 

Notations and some definitions are given. Then we will prove Theorem 1. 

Notations are as in [H; A; L; C1; C2]. To every Weil divisor D on a normal 
scheme X there is an associated reflexive rank one sheaf which we will denote by 
©,(D). The theory of sheaves of Weil divisors is developed in [R]. 


Definition 1. Suppose that f:X — Y is a proper morphism where X is a normal 
scheme. A Cartier Divisor D on X is called numerically effective for f if (D-C) = 0 
for every curve C contracted by f. 
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Definition 2. (in [ZS] Appendix 4) Let R be a normal domain and let J c R be an 
ideal. An element zeR is said to be integrally dependent on / if it satisfies an 
equation of the form 


24+ a,z7"' +a,27°7+...+4,=0 with ael'. 


The completion of J, denoted by J, is the set of all z€R which are integrally 
dependent on /. I is called a complete ideal if J = J. 
We will make use of basic properties of complete ideals in [ZS] and [L]. 


Definition 3. Let (R, m) be a complete normal local domain of dimension two. We 
will say that R satisfies condition (*) if the product J* is a complete ideal for any 
complete m-primary ideal J in R. 


Lemma 1. Suppose that R satisfies (*). Let f:X — spec(R) be a birational projective 
map such that X is normal. Suppose that D is an effective Cartier divisor with 
exceptional support for f, and that O,( — D) is generated by global sections. Then 


H°(Ox(— D)) @ H°(Ox( — D)) > H°(Ox( — 2D)) 
is a surjection. 


Proof. I = H°(©,( — D))is a complete m-primary ideal, and the completion of J? is 
H°(O,( — 2D)) by section five of [L]. 


Lemma 2. Suppose that (R, m) is an excellent normal local domain, and that I < R is 
an ideal. Let R be the completion of R with respect to m. Then I is a complete ideal in 
R if and only if I is a complete ideal in R. 


Proof. Let f:X — spec(R) be a birational projective map such that X is normal, and 
IO, is invertible. Let a, ei 
f':X' = X ® R— spec(R) 


be the natural map. X’ and R are normal. J is complete if and only if J = f, (10x) 


and [ is complete if and only if filo) =f. But fiLOx-) =f,(1Ox)@R and 
I = 1@R, hence the lemma follows since completion is faithfully flat. 


Lemma 3. Suppose that (R, m) is a normal local domain of dimension two and that 
k = R/m is algebraically closed. Suppose that X — spec(R) is a projective birational 
morphism such that X is normal. Let EX be the reduced exceptional locus. Suppose 
that C < X is an irreducible curve with exceptional support. Then there exists 
a positive integer b(C) such that if F is a line bundle on C with deg(#) > b(C), then 
£ is generated by global sections, H'(#) = 0,and Z =~ O(Q, +... + Q,) for some 
t > 0 where the Q; are nonsingular points of both X and EX. If F is a coherent sheaf 
on C then there exists a positive integer b(F ) such that if # is a line bundle on C with 
deg(L) > b(F), then H'(F ®@ H) =0. 


Proof. Given a coherent sheaf ¥ on C we will first find an integer b(F ) such that 
H'(F ® #)=O0 if F is a line bundle on C with deg(#) > b(F). Let C’ be the 
normalization of C. There is an exact sequence 


0-0,70. 7 # 0 


where # has finite support. There is an exact sequence 


0 —> Hom ¢ (#, F) > Hom 6 (O¢., F) - Hom « (Oc, F) = F — coker(v) + 0 
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Set J = Homo,(Oc, F). D is an Oc, module. coker(v) has finite support. Hence 
H'(coker(v) @ Y) = 0 for any line bundle Y. There are exact sequences 


0 — ker(v) — D — image(v) > 0 
0 — image(v) ~ F — coker(v) + 0 (1) 


There exists a > 0 such that if / is a line bundle on C’ with degc.(W) > a, then 
H}(C’, 2 ® N) = 0 by I1.5.18 and III.7.12.1 [H]. Set b(F) = a. (1) shows that if 
£ is a line bundle on C with deg(#) > b(F), then H'(F @ L) =0. 

Now we will demonstrate the existence of b(C). Let P,,..., P, be the singular 
points of EX and X contained in C. Let mp,,..., mp be their ideals sheaves. Let 


b(C) = max {b(O,) + 1, b(mp,),. . ., b(mp,), b(mp, mp, . . . mp,)} . 


If Y is a line bundle on C with deg(¥)>b(C), then H'(Y)=0 and 
H'(£ @ mg) = 0 for any point Qe C with ideal sheaf my. Hence # is generated by 
global sections. 


HL) L/(mp,mp,...mp)@® L 


is a surjection, which implies that there exists a global section o of Y whose 
support is disjoint from P, +...+ P,. Hence the divisor of o is Q, +...+Q, 
where Q; are nonsingular points of both X and EX, and Y ~ O-(Q, +... + Q,). 
We will introduce some notation for use in Definitions 4 and 5 and Lemma 4. 
Suppose that (R, m) is a normal local domain of dimension two such that k = R/m 
is algebraically closed and that f:X — spec(R) is birational and projective where 
X has rational singularities. Let EX be the reduced exceptional locus of f- 
Suppose that EX = E,uU ... VE,UC,uU... UC, where the. E; and C; are 
the distinct irreducible components of EX, sing(X)CUE;, and _ that 
sing(C;) < U E; for all j. Suppose that H is an effective Cartier divisor on X with 
exceptional support and that —H is numerically effective on X for f. 


Definition 4. A decomposition sequence for H is defined to be a datum: 
€ =(Do,...,D,) 
such that 
Do =E 
D, = E+ Cay 


D, = D,-4 + Cow = H 
Where the support of E is UE; and o:[1,2,...,r] >[I,...,s]. 


Definition 5. Suppose that @ is a decomposition sequence for H. Then we will 
define associated sheaves F; for @ and associated constants for ¢ 


v=v(@), b, =b,(@),...,b,=b,@) 
by the following construction. 


Let H = Xa,C; + XB,E;. Given pe EX such that p is not contained in any E,, 
define 
v(p) = 24,6; 
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where 
2 5 if peC,; 
‘ 10 if p is not in C,; 
Let v = v(@) = maXpeex- Vz, (v(p)). Define coherent 0; modules F; by 
F; = Oy( — Dj-1)/Ox( — D,) = Ox( — Dj-1)/0x( — Di-1 - Caw) 


for 1 Si <r. There exists positive integers b, = b(@) such that if / is a line bundle 
on C, with deg (WY) > b;,, then H'(F¥,;@ V)=0 by Lemma 3. Note that 
Ox(— D,-1 — mH)/O,( — D; — mH) =~ F,@ Ox( — mH). 


Lemma 4. Suppose that R satisfies condition (*), k = R/m is algebraically closed, 
and that f:X — spec(R) is a birational projective morphism such that X has rational 
singularities. Suppose that H is a Cartier divisor on X with exceptional support such 
that — H is numerically effective. Suppose that H has a decomposition sequence 
€ =(D, =E,...,H = D,) with associated constants v = v(@) and b; = b,(@) as in 
Definitions 4 and 5. Suppose that # is a line bundle on X such that 


(1) deg. (#) > (max, =;5;) + v for all j. 

(2) H'(O,( — mH) ® #*) = 0 for all m= 0 and k = 1 or 2. 

(3) # @ O, is generated by global sections. 

(4) H(L @0,)@ H(L ®O,) > H(L® ® O,) is not surjective. 


Then 

(A) & is generated by global sections 

(B) H°(f) @ H°(L) > H°(L®) is not surjective. 

Proof. Consider for m= 0 and WY = ¥ or #®” the following exact sequences. 
0-F, @ O;( = mH)® N > Onn +0,@ NV > Onn +2 @ NV 70 


0 > F,@ Ox( — MH) @® N > Onn + 0, @ MN > Onn + v,., ON 70 


0 F, @ Ox( — MH) @ N > Oms yp @N > Onn + v,., O@ NV 70 
0 0,( — mH) ® N > Onn. -@N 7 Ony @ N 79 


Assumption (/) of the lemma implies that H'(F, @ O,( — mH) @ /) = 0 for 
m= 0 and 1 <i<r. Assumption (2) implies that H'(0,( —mH)@ /) =0 for 
m= 0. Hence H°(Ony @ V)— H°(O, ® /) is a surjection for all m>0, and 
H°(/) = Lim H°(Ony @ MV) > H°(O, ® M) is a surjection. In particular, by 

oe 


assumption 4), H°(#) @ H°(L#) > H°(L#®”) is not a surjection, and by assump- 
tion 3), Y is generated by global sections at points contained in E. 

Suppose that pe EX is a closed point such that p is not contained in any E;. By 
assumption, p is a nonsingular point of X. Let g: Y + X be the blow up of p with 
exceptional curve L. Let C; be the proper transform of C; by g, and let H’ = g*(H). 
Identify g*(E) with E. Set a = v(p) < v. Recall that v(p) is defined in Definition 5. 
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H’ has a decomposition sequence 
Do =aL 
D, =aL+E 
D, =D, + Coa 


Di =Dj-1 + Cou-1) 
it+1 = Di + Cow 


H =D); =D, + Cow 
We have 
D, =g*(E)+aL 


D, == g*(Dj-1) + (6-1) L 
i+1 = 9*(D;))+5b,L forsome 5b; with O0<b,<Sa 


H’ =D,,, = 9*(D,) 


Set VY = g*(L). Clearly 0,,( —mH’')@ NV ~= ©, H'(O.,( — mH')@® NV) =0 
for all m, since L~P' and (L)? = —1. This shows that H°(0,,@%)— 
H°(O, ® NW) = H°(@,) is a surjection. 

For 1 Si <r define coherent ©. modules J; by 


9; ae Oy( or D;)/Oy( ~Di~ Cow) . 
Consider for m 2 0 the following exact sequences 
0 > O.( — mH’) @ N > Onn +a, @ NV > Onn @ N +0 (2) 


0-0,(— mH’ —aL)@ NV ~0,(—mH)®@ 2 > Onw +£+a.® NV > Onn +ar® NV 0 
0> FD, @O(— mH')@N > Onw + v, @NV 7 Onw +, @ NV 79 


0 9; ® Oy( — mH’) ®@ N > Onw + v;,, @ VM F Onw +, ®@ N 70 


0> F, @ Oy( — MH’) @ N > Omsi1yn @N > On +n, ® NV 70 


If p is not contained in the support of C;,, then 9; ~ F,. If p is contained in the 
support of Cj then 9; ~ F,@ Oc,,(—[bi-1]p). Hence we have inclusions 
F; ® Oy( — mH) @ Oc, ( — vp) @ # > ,@ Oy( — mH') @ WN for 1 <i <r, where 
Oc,( — vp) is defined to be ©, if p is not contained in the support of C,;. From 
assumption (/) it follows that H'(2; @ ©,( — mH’) @® WV) =0 for 1 <i<r and 
m = 0. Assumption (2) shows that H'(0,( — mH’) ® #) = 0 for all m = 0. Consid- 
eration of the exact sequences (2) now shows that H°(.”)=H°(#)—> 
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H°(0,@N)= L/m,f is a surjection, where m, is the maximal ideal of Oy, ,, and 
f is generated by global sections at p. Since # is generated by global sections at all 
points contained in EX, # is generated by global sections. 


Lemma 5. Suppose that C is an irreducible reduced (but maybe not normal) curve 
over an algebraically closed field k with h'(O,) > 0. Then there exists a line bundle 
£ on C such that 


(1) # is generated by global sections 
(2) H\(Y) = H'(Y*) =0 
(3) H°(f) @ H°(L) > H°(F2®) is not surjective. 


Proof. We will use the notation of II §7 [H]. If QC is a closed point, let mg be its 
ideal sheaf. Let Pe C be a closed nonsingular point. There are only a finite number 
of singular points Q,,...,Q, on C. If QC is not a singular point then mg is a line 
bundle of degree — 1. Hence by Lemma 3 there exists a smallest B => 1 such that 
H'(O,(BP)@®mg)=0 for all QeC. Let L=O,(BP). Clearly B>1 since 
H'(O-(P) ® mp) = H'(0 ,) + 0. First we will show that |BP| is base point free. 
Given Qe€C, there is an exact sequence 


0 > mg @ O-(BP) > O-(BP) > O-(BP)/mgO-(BP) > 0 


Hence H°(0,(BP)) > O¢( BP)/mgO-( BP) is a surjection, and O,(BP) is generated by 
global sections. 

Now we will show that there exists Q € C such that | BP| does not separate P and 
Q. We know that H'(0,(BP)® mp) = H'(0,((B — 1)P)) =0, but there exists 
a closed point QEC such that H'(O,((B — 1)P)® Mg) + 0. Consideration of 


0 > mg @ Oc((B — 1)P) > Oc((B — 1)P) > Oc((B — 1)P)/mgO-((B — 1)P)>0 (3) 


shows that H°(©,((B — 1)P)) > O-((B — 1) P)/mg0-((B — 1)P) is not a surjection. 
There is an inclusion 
Oc(BP) @ mg > O-((2B — 1)P @ mg) 


H'(O-(BP) ® Mg) =0 implies that H'(0,((2B — 1)P)®@ Mg) = 0. Tensoring (3) 
with ©,(BP) shows that |2BP| separates P and Q. In_ particular 
H°(©-(BP)) ® H°(O-(BP)) + H°(©-(2BP)) cannot be surjective since every section 
of H°(0,(BP)) containing P in its support must also contain Q. 


Proposition 1. Suppose that R satisfies condition (*), k = R/m is algebraically closed, 
and f :X — spec(R) is birational, projective such that X has rational singularities. Let 
EX be the reduced exceptional locus of f. Suppose that EX = CUC,... UC, where 
C, C,,...,C, are the distinct irreducible components of EX. Suppose that 
sing(X) < C and that sing(C;) < CAC; fori=1,..., e. Then H*(@,) = 0. 


Proof. Suppose that H'(0,) + 0. Let b = b(C) be the number computed in Lemma 
3. There exists a smallest integer ry > 0, such that roC is a Cartier divisor on X. 
Let «:Z-—X be a resolution of singularities. Let C” be the proper transform 
of C by a Then dege(Oc(roC)) = dege-(Oc-(g*(roC))) = (C”*(9* PoC))) 


1 ; ; ; ; F 
= *)agrrscy? <0 since the intersection form for Z—spec(R) is negative 
To 

definite. Given n 2 0, write n= mr, + t withO St <rp. 


Ox( — nC)/Ox( — (n + 1)C) ~ [Ox( — t€)/Ox( — (t + 1I)C)] @ Oc( — mroC) 


Set d = max {b(C), b(Oy( — C)/Oy( — 2C)),. . ., b(Ox( — (Fo — 1)C)/Ox( — roC))}- 
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By Lemma 3, if # is a line bundle on C with deg(#)>d _ then 
H'(# @ [Ox( — nC)/O,( — (n + 1)C)]) = 0 for all n= 0. 

Let y:X’ — X be the blow up of general closed points W,,..., Wi4, of C. Let 
C’” = C be the proper transform of C by y. 


Ox (— nO" Ox — (n + 1)C") 
~ [Ox( — nC)/Ox( — (n + 1)C)] @ Oc(n(W, +... + Wass) 


Hence if # is a line bundle on C” such that deg(#)20 then 
H'(# ® [Oy — nC’”)/O,.( — (n + 1)C’”)]) = 0 for all n > 0. Without loss of gen- 
erality we may replace X by X’. If # is a line bundle on C such that deg,.(#) = 0, 
then 


H'([O,( — nC)/O,(—(n + 1)C)]® #) =0 forall n>0. (4) 


Let O,(BP) be the divisor constructed in the proof of Lemma 5. There exists 
A > 0 such that 


b —(C-rgC) 2 A(C-(—1roC)) — B>b 
Set r, = Ary. Let s = A(C-( — rgC)) — B. There exists H’, an effective ample Cartier 
divisor on X with exceptional support such that 
(C:(—H’))>b (5) 
and H’ has a decomposition sequence ¢’ = (Do,..., D;) satisfying the assum- 
ptions of Definition 4, and which can be written as (6) below. 
Do =aC (for some a > 0) 
Di = Do + Cay 
(6) 


H’ = D, = D,-1 + Cow 


Let F; be the associated sheaves of @’ and b; = b,(@') and v = v(@’) be the 
associated constants for @’ computed in Definition 5. There exists an effective 
Cartier divisor 4 on X with exceptional support such that (4:C)=0, and 
(C;( -A- r,C)) > (max, =; 5)) + v. 

By Lemma 3, there exist nonsingular points Q,,...,Q, of both X and EX such 
that O,.(— 4 — BP —1r,C)~ O,(Q, +...+@Q,). WriteQ, +...+0,=Yi-, 4,0, 
where t Ss, Xd; = s, and Q, + Q, if i +j. 

Let gY — X be the blow up of Q,,..., Q, with exceptional curves F,,..., F,. 
Let C; be identified with their proper transforms by g. Let C’ be the proper 
transform of C by g. Let 


D = g*(— 4) —g*(r,C) —d,F, —...—4,F, 


D is Cartier and has exceptional support for Y—spec(R). By construction, 
Oy(D) ®@ Oc. = O-( BP). (D+ F;) = d; = 1, and 


(D-C) =((= 4 =.0)-C,)>( max by) +0. (7) 
a(j)=i 


for all i. 
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Let H = g*(H’). H has a decomposition series @ = (Dy, .. ., D,) as in Defini- 
tion 4, obtained from (6) by applying g*. @ can be written as: 


Do = 9* (Do) 
D, =g*(D,) 


H = D, = g*(D,) 


In the notation of Definition 5, g*(F,),. . ., g*(F,) are the associated sheaves 
of @, and we can take b; = b,(@) = b,(@’) for 1 Si <r, v = v(@) = v(@’) to be the 
associated constants for @, since g is an isomorphism in a neighborhood of u C;. 

Set E=D,)=a(C'+F,+...F,). Let # =0,(D). We will show that the 
decomposition sequence @ with its associated constants b; and v, and ¥ verify 
assumptions 1)—4) of Lemma 4. Assumption /) is verified in equation (7) above. 

Consider for m= 0,a=>n=0 and WV = ¥ or Z® the sheaves 





: O,(—n(C' + F, +...+F,)) 
Fmn) = j| renee +...+F,)) 


There are exact sequences: 


|e-reo—min}. 


0 Of, 4...47(-C’ —n(C'+F, +...+F))@NV @O,(— mH) 
On +1(C'+F,+...+F) @ MV @ Oy( — mH) > 
Oc +c’ +F,+...4F)@M @ Oy( — mH) > 0 
0 F(m, n) > Oc 4 wo+F, +...+7) OV @ Oy( — mH) > 
> Onc +F,+...F)@ NM @ Oy( — mH) 0 
h' (Op, 4...42(—C’ —n(C' + F, +...+F,))® WN @ O,( — mH)) 
= ¥h'(Op( — 1) @ Gp(ed,)) = 0, where c=1 or 2 
i=1 


depending on whether or not W = ¥ or #®. 


F (m,n) = [Oy( — nC’)/O,( — (n + 1)C’)] 
@ O,( — n(F, +... + F,)) @ NV @ Oy — mH) 
~ [O,( — nC)/O,( — (n + 1I)C)] @ VY @ O,( — mH’) 


When n > 0, H'(F (m, n)) = 0 by (4). A(m, 0) = 0. @ NM @ Ox( — mH’). Hence (5) 
shows that H'(¥(m, 0)) = 0 for m> 0. F(0,0) = 0. @ MY. H'(0,@® NW) = 0 and 
Oc ® MN is generated by global sections by the choice of Y. Assumption 2) of 
Lemma 4, and the fact that # @ ©, is generated by global sections in a neighbor- 
hood of C’ now follows from induction using the sequences (8). Assumption 4) of 
Lemma 4 is true by Lemma 5, since H°(O, ® V’) > H°(@- @ ) is a surjection. 

It remains to verify that 7 ® 0, is generated by global sections in a neighbor- 
hood of each F;. We need only check for F, , since the argument is the same for each 
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F,. For m 2 0, set 
K,(m) = [0,(-— F, —m(C' + F, +... + F,))/Oy(— F, —m(C'+ F, +... 
+F)-C')]@# 
K, = O7,4...4rA(-C—F,-m(C' + F, +...+F)) Of 
K,=O;(—(m+1I(C'+F,+...+F)) @F# 
There are exact sequences: 
0> K,(m) > Oc 4 5,4 mc+r,+...+F) @L Or, 4 mc+F,+...4F-)8Lf 70 
0 K, > Om 4 yc4F,+...+F) @ LF Oc 4 6, 4mC+F, +...4F)8L 70 
0+ K3 7 Oms ctr, +...+F)+F, OF 7 Ome c+ F, +...4F-) OL 70 (9) 
K,(m) = [Oy( — mC’)/Oy( — (m + 1)C')] @ O/(— m(F, +... + F,)- Fi) @L 
~ [Ox( — mC)/Ox( — (m + )C)] @ O¢( — Q, + BP) 


H'(K,(m)) = 0 for m > 0 by (4). H'(K,(0)) = H'(O,(BP — Q,)) = 0 by choice of 
LY. K,~ @i-20p(d; — 1). Hence H'(K,)=0. K3;~Op(d,) implies that 
H'(K,) = 0. In particular, consideration of the exact sequences (9) and induction 
shows that H°(0,®@ %) > H°(Or,® L) = H°(Op(d,)) is a surjection, and 
O, ® & is generated by global sections in a neighborhood of F,. 

Now Lemma 4 shows that H°(#) @ H°(#) > H°(F®”) is not a surjection, in 
contradiction to Lemma 1. This contradiction shows that H'(0,) = 0. 


Proposition 2. Suppose that R satisfies condition (*), k = R/m is algebraically closed, 
and that f :X — spec(R) is a birational projective map where X has rational singular- 
ities. Suppose that EX is the reduced exceptional locus of f, and that C is an 
irreducible component of EX such that sing(X) < C. Suppose that if A < EX is an 
irreducible component, then sing(A) < C. Then C x P', and 0,( — C)/O,( — 2C) 
~ Op:(e) for some e with e = — 1. 


Proof. C = P' by Proposition 1, consideration of the normalization sequence 


0-0,.70,. 7 F 70 


where C’ is the normalization of C and F has finite support, and by IV.1.3.5 [H]. 
O,(—C)/O,(—2C) is a line bundle on C since C is nonsingular and 
O,( — C)/Oy( — 2C) is torsion free. 

Suppose that 0,( — C)/Oy( — 2C) = Op:( — s) with s = 2. We will derive a con- 
tradiction. Let «:X’— X be the blow up of s+3 general points on C with 
exceptional curves F,,..., F,, F,+,, L,, L,. Let F be the proper transform of C by 
a. Then O,y.( — F)/Oy.( — 2F) = Op.(3). 

Let B’ = 2F + F, + F, + F; +L, + L,. Let # be a numerically effective line 
bundle on B’ such that deg, (#) = 1 for some i. We will show that H'(#) = 0. 


Without loss of generality, we may suppose that deg, (Y) > 0. There are exact 
sequences 


0 [Ox — F)/Oy( — 2F)]@ O,( — F, — F, — F; —L, )@# 
= Op:( — I @L > Ox. 646,46, 41,0L > 
OrsF,+F,+F,+1,0f 70 
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0-0, (-—2F — F, — F, — F; —L,)@# = Op(—-27@L— 
L > Ow 47.47, 4+F,+1,0Ff 70 (10) 


It is straight forward to check that H'(Or,r,4r,+F,+1,®L)=0, hence 
H' (Og @ L) = 0. (10) also shows that if deg, (#) = 1 for i = 1 and 2, then # is 
generated by global sections. A similar sequence shows that H'(@,.) = k. 

Let a, and a, be general (closed) points of L, and L, respectively. There exists 
a Cartier divisor D’ with exceptional support for X’ > spec(R) such that (D’- y) = 0 
if y is exceptional and y + L, or L,, but (D’-L,) > 0 and (D’-L,) > 0 since the 
intersection form for X’ is negative definite. 

Let z, = (D’:L,) — 1, z, =(D’- L,) — 1. Let B:X” > X’ be the blow up of a, 
and «, with exceptional divisors A, and A,. Let D” = B*(D’) — z,A, — 2,A,.D" is 
numerically effective on X” with exceptional support for X” — spec(R). Identify F, 
F,, F,, F;, L, and L, with their proper transforms by f. (D”-L,) =(D"-L,) = 1, 
and (D” - F) = (D”: F,) = (D" - F,) = (D” : F;) = 0. 

Let y:Z — X” be the blow up of general (closed) points on F,, F,, F;, L,,and L, 
so that if F, F,, F,, F;, L, and L, are identified with their proper transforms by y, 
then there exists t>0 such that if W=tF + F, + F,+F,+L,+L,, then 
W is Cartier, (— W-F)>0, (— W-F,)>0, (—W-F,)>0, (— W-F;)>0, 
(— W-L,)>0, and (— W-L,)>0. Let B= 2F + F, + F, + Fz + L, + L,. Let 
D = y*(D"). Let H be an effective Cartier divisor on Z with exceptional support for 
Z-—spec(R) such that —H is ample. H has a decomposition sequence 
€ =(Do,..., D,) as in Definition 5, such that @ can be written as: 


D, = E+ Coy 


H => D, = D,-1 + Cow 


where E is an effective divisor such that sup(E) = FU F, UF, UF, UL, UL, and 
{C;} are the irreducible components of the exceptional locus of Z — spec(R) not 
contained in sup(E). Let b; = b,(@) and v = v(@) be the associated constants of @ as 
defined in Definition 5. 

Let A be an effective Cartier divisor on Z with exceptional support such that 
(4-F) =(4-F,) =(4-F,) = (4-F) = (4-L,) =(4-L,) =0, and 


deg.,0c(— 4) > ( max b) +v (11) 


a(i)=j 


for all j. 

Let Y = 0,(D — A). There exists b > 0 such that bW > E. By replacing H by 
a sufficiently large multiple of H we may assume that E > B. We will show that the 
decomposition sequence @ with its associated constants b; and v, and # satisfy 
assumptions (/) through (4) of Lemma 4. Assumption (2) follows from (11) and the 
fact that D is numerically effective. Let “” = Y or #®. We will show that 


H'(O,w ® VY @,(—mH))=0 for m2=0 (12) 
and the natural map 


H°(Osw ® NV @ O;( — mH)) > H°(0, @ NV @ O;( — mH)) (13) 
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is a surjection for m= 0. For OS a<b and OS n<t-—1 there are exact se- 
quences: 
0— [0,( — B — nF)/0,( — B — (n + 1)F)] @ O;(— aW) @ NV @ O;( — mH) > 

Op+nt+1)F+aw @ V @ O2(— mH) 

Op+nr+aw @ NV @ O;(— mH)>0 (14) 
0 0;(-— aW)@ NV @ O,( — mH) > Of+a" ® NV @ O;( — mH) > 

Ow ® NV @ 0;( —mH)-0 

[0,( — B — nF)/0,( — B—(n + 1)F)]® 0,( — aW) @ WV @ O,( — mH) 

~ [O,( —(n + 2)F)/O;( — (n + 3)F)] @ O;( — F, — F, — F; — L, — L) 

@ 0,(-—aW —mH)@®N 
~ [0;( — (n + 2)F)/Oz( — (n + 3)F))] @ Op:( — 5) @ O;( -— aW — mH) @ V 


There exist A = 3(n + 2) — 5 = 1 such that there is an inclusion 


Op(A) = [0z( — F)/Oz( — 2F))]®"*” @ Op:( — 5) @ O;( — aW — mH) @ VW > 
[0z( — (n + 2)F)/Oz( — (n + 3)F)] @ Op:( — 5) @ O,( — aW — mH) @ N 


Hence H'((0,( — B — nF)/O,( — B —(n + 1)F)] ® O,( — aW — mH) @ /) = 0. 
H'(0, @ 0,( — aW — mH) @ /) =0 since 0, @ 0,(-—aW — mH)® J is nu- 
merically effective and deg, (O;( — aW — mH) @ /) 2 1 for i = 1 and 2. (14) now 
shows that formulas (12) and (13) are true. 

There is an exact sequence 


0-K-Oyw@NV-0,@N 0 
where K is defined to be the kernel. This induces an exact sequence 
0 = H' (Ow ® NV @ O;( — mH)) > H' (0, @ NY @ O,( — mH)) > 
H?(K @ ©,(— mH)) =0 


Hence H'(0, ® VY ® ©,( — mH)) = 0 for all m = 0, and assumption (2) of Lemma 
4 is verified. 
Since the composition 


H°(Opw ® V) > H°(0, @ NV) > H°(0,® Y) 


is surjective, 


H°(0O,® NV) > H°(O, ® ) is surjective, (15) 


and 0, @ £ is generated by global sections since H°(O,@ L) is generated by 
global sections. This verifies (1), (2) and (3) of the assumptions of Lemma 4. 
It remains to verify assumption (4). 


deg, (Y) = deg, (Y)=1, and deg,(#) = deg-(L) =0 


for all i and the fact that “ @ ©, is generated by global sections implies that 
L# @ O; = O;(P + Q) where P and Q are points of L, and L, respectively, which 
are nonsingular points of B. We know that H°(Y @ O,) is base point free. We will 
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show that it does not separate P and Q. Consider the exact sequence 
0 + Os; > O,(P) > Op(P)/mpO,(P) > 0 
where mp is the ideal sheaf of P. This induces an exact sequence 
H°(Os(P)) > Og(P)/mpO(P) = k + H*(Oy) = k + H4(O,(P)) = 0 


which shows that H°(Y @ O,) does not separate P and Q. But tensoring (16) with 
£ and taking global sections shows that #® does separate P and Q. Hence 
H'(L @0,)@H(L@0;) 7+ H(L”*@Oz) is not surjective. By (15), 
H°(L @ 0,)@ H'(L ® O,) > H'(L® @ O,) is not surjective. 

Now by Lemma 4, H°(#) @ H°(#) > H°(F®”) is not surjective. The exist- 
ence of such an ¥ gives a contradiction by Lemma 1. 


Proposition 3. Suppose that R satisfies condition (*), k = R/m is algebraically closed 
and that f:X — spec(R) is a resolution of singularities such that f factors through 
Y and that Y has rational singularities, where h and g are birational projective 
morphisms. 


h 
xX — ; aes spec(R) 


Let F be an integral curve which is exceptional for g. Let E be the proper transform of 
F by h. Suppose that sing(Y) < F and that the singular locus of each exceptional 
curve for g is contained in F. Then deg((I;0x) ® Og) = — 1. 


Proof. By Propositions 1 and 2, E ~ F ~ P’. By Proposition 2 of this paper and 
Lemma 4 of [C1], 


— 1 S deg,(©,( — F)/O,( — 2F)) = deg-(I,/I!?) 
= deg,((Ip-Ox) @ Og). 


Proposition 4. Suppose that R satisfies condition (*), k = R/m is algebraically closed, 
f.X — spec(R) is a resolution of singularities which factors through Y, and that Y has 
rational singularities, where h and g are birational projective morphisms. 


h 
X —— Y—+spec(R) 


Let F be an integral curve which is exceptional for g. Let E be the proper transform of 
F by h. Suppose that sing(Y) < F and that the singular locus of each integral 
exceptional curve for g is contained in F. Then deg((I 0x) @ O,) = — 1 for all 
n>0. 


Proof. The proof of Proposition 3 [C1] with references to “Proposition 3” of [C1] 
changed to “Proposition 2” of this paper, proves this proposition. 


Proposition 5. Suppose that (R, m) satisfies condition (*) and k = R/m is algebraic- 
ally closed. Then R has rational singularities. 
Proof of Proposition 5. Let X — spec(R) be a resolution of singularities such that 


every integral exceptional divisor is nonsingular. There exists a factorization of 
birational projective morphisms 


x pa h, 
X= X_..n0},. io; . elie ae 
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Where each h, contracts a single integral exceptional curve E,, X, is normal, and 
the singular and fundamental locus of X, is contained in E, by Theorem 2 of [C2]. 
Let a’: X — X, be the induced maps. 


We will show that R'a’~'O, =0 for all r by induction. For r=n + 1 it is 
trivial so we will suppose that R'a’,O, = 0, and show that R'a,~'O, =0. Let 
E, be the proper transform of E, by a’. X, has rational singularities, so that 
dege((If Ox) @ Og) = —1 for all s20 by Proposition 4. By Lemma 5 [C1] 
X,-, has rational singularities. 


Theorem 1. Suppose that (R, m) is an excellent and normal local domain of dimension 
2. Suppose that k = R/m is algebraically closed. Then the following are equivalent. 


(1) R has a rational singularity. 

(2) If I and J are any two complete ideals in R, then the product IJ is a complete ideal. 

(3) If I and J are any two complete m-primary ideals in R, then the product IJ is 
a complete ideal. 

(4) If I is any m-primary complete ideal in R, then I? is a complete ideal. 


Proof. (1) implies (2) is Theorem 7.1 of [L]. (2) implies (3) and (3) implies (4) are 
obvious. 

It remains to show that (4) implies (1). Let R be the completion of R with 
respect to m. Then the m-primary ideals of R are the extensions of m-primary ideals 


of R. By Lemma 2, we may assume that R is complete. Theorem 1 then follows from 
Proposition 5. 


II. It is necessary that k is algebraically closed 


In this section we give a proof of Theorem 2. 

Theorem 2. Let k be a field of characteristic not equal to 3. Let R(k) be the ring 
R(k) = k[ [x, y, z]]/x? + 3y? + 92°. 

The following is true 


(1) R(k) is anormal local domain of dimension two and a simple elliptic singularity. In 
particular, it is not rational. 

(2) Let k = Q be the rational numbers. If I, J < R(Q) are any two complete ideals, 
then IJ is complete. 

(3) If k has positive characteristic, or if k is an algebraically closed field, then there 
exists an m-primary complete ideal I < R(k) such that I* is not complete. 


Let C = proj(k[x, y, z]/x* + 3y? + 923). C has a natural very ample divisor (1). 
C is an elliptic curve. 


We will first establish (/) of Theorem 2. By the Jacobian criterion it can be seen 
that R(k) is a normal domain. Let 1: X > spec(R(k)) be the blow up of the maximal 
ideal. By direct calculation (V.2.11.4 [H]) it can be seen that A is a resolution of 
singularities, the exceptional locus of 4 is C, Ox(—C)@®0O,.~O,(l), and 
H'(0,) =k, so that R(k) is a simple elliptic singularity. 

Now we will establish 2) of theorem 2. Assume that k=Q, so that 
C = proj(Q[x, y, z]/x? + 3y? + 923). 
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Lemma 6. The image of deg: Pic(C) > Z is 3Z. 


Proof. Suppose that C has a line bundle L of degree 1. Then by The Riemann-Roch 
theorem, h°(L) = 1 since C is an elliptic curve. Hence L ~ ©,(p) where p is 
a rational point of C. This means that there is an integral solution in x, y, z, other 
than x = y = z = 0, to the diophantine equation 


x? + 3y? +927 =0. 


By the fundamental theorem of arithmetic this is impossible, since 3 must divide 
any solution x, y, z infinitely many times. Hence C does not have a line bundle of 
degree 1. 

Suppose that L is a line bundle of degree 2. Then deg(0,(1) ® L~') = 1, which 
contradicts the above paragraph. 


Proposition 6. Suppose that L and M are line bundles on C such that L and M are 
generated by global sections. Then H°(L) ® H°(M) > H°(L @ M) is surjective. 


Proof. We may assume that LZ and M are not isomorphic to ©,, and that 
deg(L) = deg(M). First assume that LZ is not isomorphic to M. Then 
h'(M~! @ L) = h'(L) = 0 by the Riemann-Roch theorem. Let o¢ H°(M). H°(M) 
is generated by global sections, hence there is a surjection (0,.)' + M. Define F by 
the exact sequence 


0+M-!—+0,+F +0 


0+>M'@L@OL>L@Ob F@LQ@Ot 
J | i 
0 ad : L@M—F@L@M-—0 





This induces an exact sequence 
0 H°(M~' @ L)@ H°(M) > H°(L)@ H°(M) > H°(F @ L)@ H°(M) +0 


Vv v2 Vs 





0 H°(L) a »H°(L @ M)—> H°(F ® L® M) > 0 


W, is surjective since the kernel of the surjection (0,)' @ L@ F+>M@L®@F 
has finite support. 

Clearly cH°(L) is contained in the image of y,. Hence yw, is surjective. 

Now assume that L~M. First suppose that L~O,(r). Then 
H°(Op:(1))®! + H°(p.(i)) is a surjection for all i > 0, and H°(Op:(i)) + H°(O,(i)) is 
a surjection for all i => 0. Hence H°(L) ® H°(M) > H°(L @ M) is a surjection. 

The last case is when M =~ L = 0,(r) @ N where N is a divisor of degree 3 on C, 
and N is not isomorphic to @,(1). First suppose that r > 0. By the previous 
analysis, we know’ that H°(M)@H°(0,(r))>H°(M@QO,(r)) and 
H°(M @ O-(r)) @ H°(N) + H°(M@O-(r) @N) are ___ surjective. Hence 
H°(M) @ H°( @-(r)) @ H°(N) > H°(M) ® H°(L) > H°(M @ L) is surjective. 
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Suppose that r = 0. Then since M has degree 3, the Riemann-Roch theorem 
shows that there is an embedding g:C > P? such that g*0(1) ~ M = N. Hence 
H°(M) @ H°(N) > H°(M @ N) is surjective. 


Lemma 7. Suppose that £ is a line bundle generated by global sections on X..Then 
either deg-(# ® O,) = 3r withr > 1 0r Y= Oy. 


Proof. deg-(# ® O-) = 3r with r 2 0. For all n 2 1, 

0 > O¢( — nC) > Of14 1c > Onc 7 9 
is exact. h'(0,(—nC))=0 for all n2=1. Hence Pic(X)= lim Pic(nC)= 
jim H 1(0*.) = H'(08). If Y is generated by global sections, and if deg(¥ )=0, 
then £ @ 0, = O-. But then # =~ Oy. 


Lemma 8. Suppose that ¥ is a line bundle on X and that deg-(#) > 0. Then 
h'(#) =0. 


Proof. Consider the exact sequences 


0+ 0,8 £ @Ox( — nC) > Ons 10 @ F > Oc @ L 0 
to see that h1(0,- ® #) = O for alln = 1. Hence H'(¥) = lim H'(0,c @ L) = 0. 
— 


Proposition 7. Suppose that Y and AN are line bundles on X such that Y and N are 
generated by global sections. Then H°(L) @ H®°(V) + H°(L @ N) is surjective. 
Proof. By induction, we will show that 

H°(N)@ H°(L @ Onc) = H(N ® L @ Onc) is surjective (17) 


for all n=1, from which the theorem follows. We will first show that 
H°(V)® H°(L ® 0-) > H(N ® £ ® Og) is a surjection. Exactness of 


0+ 0,(-C)@N #>N 3-N @0, 30 


implies that H°(.W) > H°(W @ @,) is a surjection by Lemma 8. Now the surjectiv- 
ity follows from Proposition 6. 

Assume that the map (17) is a surjection for some n 2 1. We will show that it is 
a surjection for n + 1. There exists a surjection R' + H°(./) of R modules. There is 
a diagram: 


0> 0-( — nC) @ £ @ Of > Ons 10 @ FZ @ Of > Onc @ F ® OF 30 
{ { { 
0+ 06 -—0C)@ LON > On+1c® LON 7 Onc @LZON 0 
Taking global sections yields: 
0+ H(O(-nC)@L)@R + HOnsyc@L)@R' + HOc@L\@R' +0 
| | { 
HO —nC)@L)@H(N)  H°Omsryc@L)@H(N) HO ®@ L)@H (NV) 
| | | 
0+ H°Ol(-—nC)@LON) + HOmsyc@LlON) + H(Oc@Ll@N) +0 
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The right and left two vertical maps are surjective by induction, hence the middle 
map is a surjection by the five lemma. 

Suppose that h:Z — spec(R) is projective, birational, and Z is nonsingular. 
Since X — spec(R) is the minimal resolution of singularities, there exists a projec- 
tive map a:Z — X factoring h. Let C’ be the proper transform of C by «. By 
Theorem 2 of [C1], there exists a projective birational map f:Z — Y where Y is 
normal and the exceptional locus of fis C’. Let g: Y + spec(R) be the induced map. 


Lemma 9. (1) Suppose that N is a line bundle on Z such that V ® Oc: = O¢.. Then 
N = f*(#) for some line bundle X on Y. (2) H'(Oy) = 0. 


Proof of (1). Let p=f(C’), the singular point of Y. Let Z’'=Z®@ Oy.» where © 
denotes completion along p. By consideration of 


0> 0¢(— nC’) > O%+ ye —> OX: +0 


we have that Pic(Z’) ~ Pic(C’). YW @ Oc = Oc implies that VW ~ f*(#) for some 
line bundle # on Y by Lemma 5 of [C2]. 


Proof of (2). (— C’:C') 2 3 and C’ an elliptic curve shows that R'f, Oc-( — nC’) = 0 
for all n > 0. The exact sequences 


0 > O¢( — (n + INC’) > Ons tye — C') > One (— C') + 0 


imply that R'f,O,c(—C’) =0 for all n>0. Hence (R'f,0,( — C’)) ® Oy, )= 
R'f,0z(—C’') = Lim R'f, Onc (— C’)=0. Since @Oy,, is faithfully flat, 
——_ 


R'f,0,(—C’) =0. 


0- 0,(-C’) 7 0,70, 70 


now shows that R'f, 0, = R'f,0c = H'(Q,). A similar argument applied to 
X and C shows that R'A, Oy = H'(0,). But R'h, 0, = R'A, O, since X and Z are 
nonsingular. Since R'f, Oz has finite support, g, R'f, Oz = H'(O-) = R'h, Oz. By 
the Leray spectral sequence, the following sequence is exact. 


0- R'g,(Oy) = R*h, (Oz) = 9,R'f,0z a R7g,0y =0. 
Hence H'(0,) = 0. 


Lemma 10. Let notation be as in the paragraph preceding Lemma 9. Suppose 
that N is a line bundle on Z such that N is generated by global sections, and that 
H is a numerically effective line bundle on Y. Then H°(V)@H() 
+ H°(V @f*(X#)) is a surjection. 


Proof. We will use Lemma 7.3 of [L]. We need to show that f,(/) and © are 
generated by global sections to apply the lemma. By Theorem 12.1 of [L], and (2) 
of Lemma 9, # is generated by global sections. f,(”) =f, ((h,(Y )Oy)0z) 
= h, (NV) Oy. But h,(4)Oy is complete by Proposition 6.5 [L] and (2) of Lemma 
9. Hence f,(VW)=h,(V)Oy, and f,(%) is generated by global sections. 
H(N)@H(H)=9( i V)@GH. HN @f*H)=9,(f (VY @f*X)) 
= 9,(f,(VY)@#) by the projection formula. By Lemma 7.3 of [L], 
IuflV)@ G(X) > Gu Se(M) ® H) is surjective. 


Proposition 8. Suppose that h: Z — spec(R) is a resolution of singularities, and that 
£ and WN are line bundles on Z such that and N are generated by global sections. 
Then H°(L) @ H°(V) + H°(L @ ) is a surjection. 
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Proof. We will use the notation introduced before Lemma 9. Let 7’ = 7 @ O¢. 
£ generated by global sections implies that ’ = O,. or deg(#’) = 3r with r 2 1. 
There exists a line bundle #, on X such that Y,@ 0, = #’. L; is necessarily 
generated by global sections. Let Y, = a*(¥;). Let Y, = L@L;". Clearly 
deg,(£,) = 0 for every exceptional curve E of « since ¥ is generated by global 
sections and deg,(¥,) = 0 unless E = C’. #, @ O-. = Oc: by construction, hence 
L, = f*(L,) where F, is a numerically effective line bundle on Y by (/) of Lemma 
9. Hence we have a decomposition Y ~ #, ® Y, ~ a*(L;) @f*(L,) where 
L, is generated by global sections, and #, is numerically effective. There is a 
similar decomposition WV ~ NV, @NV,~a*(V3)@f*("Y,). We have that 
H°(£,)@ H(N,) > H°(L, ® N,) is a surjection by Proposition 7. By Lemma 
10, H(L,)@H (Nz) > H(L,@N2) and H(%,@N,)@H(L,@ AN) 
+ H(L @S) are surjective. Hence H°(¥,)@ H°(1,) @ H°(L,) @ HV) 
+ H°(L)@ H(V) > H°(L @ /) is surjective. 


(2) of Theorem 2 follows directly from Proposition 8. Let J and J be complete 
ideals in R. There exists a resolution of singularities h: Z — spec(R) such that IO, 
and J; are line bundles which are generated by global sections. Set Y = 10, and 
N = JOz. Then H°(IJO,) = IJ = IJ by Proposition 8. 

When k is algebraically closed, (3) of Theorem 2 follows from Theorem 1. 
Assume that k has positive characteristic. Then X has a line bundle ¥ of degree 
2 by Corollary 5 to Theorem 2 Chap. VII Sect. 5 [W]. In fact, there exists a closed 
point P of degree 1 on C such that Y = O,(2P). £ is generated by global sections 
but is not very ample, and #®? is very ample. Hence H°(#)® > H°(#®”) is not 
a surjection. 0,( — C) @ O-( — 2P) = O-(Q) where Q is a closed point of degree 
1 on C. Let g: Y> X be the blow up of Q with exceptional divisor F. Let C’ =~ C be 
the strict transform of C by g. Let D=g*(—C)-—F. By construction, 
O,(D) @ Oc =~ F.It can be checked that ©,(D) is generated by global sections, and 
H°(0,(D)) > H°(@¢(D)), H°(Oy(2D)) > H°(O¢-(2D)) are surjections. 
H°(0,(D))® + H°(O,(2D)) is not a surjection since H°(#)®? + H°(H®) is not 
a surjection. Set J =H®°(Oy(D)). I is a complete m-primary ideal, and 
I? + I? = H°(0,(2D)). 


III. An example 


In this section we will construct the following example. 


Example. There exists a normal but not Cohen-Macaulay scheme Z obtained by 
blowing up an m-primary ideal J in R=C[[x,y,z]] with natural map 
h:Z — spec(R) such that R'h, 0, =0 and R*h, 0, +0. @,zol” is normal but 
not S,. 


We will first need some simple results on projective surfaces. Let P? be complex 
projective space. Let W be a general quartic in P?. Let C” be a nonsingular cubic 
curve in P*. Let H’ be a general linear hyperplane section on P?. 
W-C" =P, +...+ P2, twelve points on C”. Let 4: E’ > P? be the blow up of 
P,,..., P;2 with exceptional curves e,,..., @,2. Let h = 4*(H’), and let C’ be the 
proper transform of C” by 4. C’ is linearly equivalent to 3h — e, —... — e;2. Let 
D=h+C". 
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Lemma 11. Let notation be as in the above paragraph. Then |mD| is base point free 
for allm > 0. There exists ny > 0 such that for all m > 0,|mn,D| induces a map @ on 
E’ such that ¢ is an isomorphism away from C’, and @ contracts C’ to a normal point. 


Proof. By construction, (D-C) > 0 for any irreducible curve C on E’ except for C’. 


(D-C’)=3+(C’)? =0. 


For all m > 0 Bs|mD\, the base locus of |mD|, is contained in C’. Consider the exact 
sequences 


0 > Og (mD — C’) > Og (mD) > O¢- (mD) > 0 (18) 


We will show that H'(0,.(mD — C’)) = 0 if m= 1. The first step is to observe 
that H'(0,-(rh)) = 0 for all r => 0. We have that (h-C’) = 3 and (C’-C’) = — 3. 
Consider 


0 O;-(rh + (m — 1)C’) > Og (rh + mC’) > O¢-(rh + mC’) +0 (19) 


((rh + mC’): C’) = 3(r — m) implies that H'(@,.(rh + mC’)) = Oif r > m, since C’ is 
an elliptic curve. (19) then shows that H'(O,(rH + mC’)) =0 if r>m2=0. In 
particular, H*(O,-(mh + (m — 1)C’)) = H'(O,-(mD — C’)) = 0. 

O¢(D) = Oc by construction. Hence in (18), H°(Og-(mD)) > H°(@¢-) is a surjec- 
tion, so that Bs|mD| = @ for m2 1. 

Since (C: D) > 0 for every curve C on E’ except for C’, and (C’- D) = 0, we have 
that C’ is the only exceptional curve for |mD|. The existence of an ny with the 
desired property follows from pp. 581-582 [Z]. 

Now we can construct the example. Let Y = spec(R). Let f,:X — Y be the 
blow up of m = (x, y, z) with exceptional surface Ey ~ P?. We will use the notation 
preceding Lemma 11. Let f,:X — X, be the blow up of W-C” = P, +...+ Pi2 
with exceptional surfaces E,,..., E,2. Let f:X — Y be the natural map. Let E’ be 
the proper transform of Eo. E’ is the surface constructed in the paragraph preced- 
ing Lemma 11. Let H=—E’—E,—...—E,2 so that O,(H)=m0,. 
K, = —2H+2E,+...+2E,;2 is the canonical divisor of X. Let 
D' = 4H — E, —...— E,2. D’: E' = D, the divisor constructed in the paragraph 
preceding Lemma 11. For each i, Og, ® ©,(D’) = Op:(1). 

Suppose that an irreducible curve C is supported in E’ + E, +... + E,2.Then 
(C-D’) > 0 unless C = C’, and (C’: D’) = 0. 


—EF'=H+E,+...+£E,, 
— 2E’ — K, = 2H + 2E, +...+2E,, + 2H —2E, —...—2E,,=4H 


Hence ©,(mD’ — 2E’') = O,(mD’ + 4H + Ky). Form = 0, H'(O,(mD’ — 2E’)) = 0 
by the Kawamata-Viehweg Vanishing Theorem, Theorem 1.2 [KMM]. Hence 
there is a surjection H°(O,(mD’)) > H°(02,-(mD’)). There is an exact sequence 


0 O;(mD’ — E’) > ©2¢-(mD’) > O;-(mD’) +0 


By the adjunction formula, O;(K,-) = Og(Ky + E’). Hence O,(mD’ — E')= 
O;(mD’ + 4H + K,,.). The Kawamata-Viehweg Vanishing Theorem implies that 
H'(O,(mD — E’))=0. Hence H°(O,(mD’))— H°(Og(mD)) is a_ surjection. 
H°(@z(mD)) is generated by global sections by Lemma 11. Hence 
Bs|mD| < u (E; — e;) which implies that Bs|mD| is a finite number of points. By 
a simple modification of Theorem 6.1 [Z2], there exists m) >0 such that 
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Bs|m,D| = @. Let A= Pn zo f,.Ox(mD). A is normal and a finitely generated 
R algebra. 

Let Z = Proj(A). Z is the normalization of the image of |m,D|. Let g and h be 
the natural maps defined by the following diagram. 


There is an exact sequence obtained from the Leray spectral sequence 
0- R'h, 0, > R'f, Oy +h, R'g, 0, > R*h, 0, > R*f, Oy (20) 


R'f, Oy = Rf, 0, = 0 since X and Y are nonsingular. g is an isomorphism away 
from C’, and C’ is contracted to a normal point Q on Z. Hence R'g, 0, is 
supported at Q. C’ is an elliptic curve so that R'g,0c¢ = H'(0c) =C. The 
exactness of 


R'g,0x > R'g,0c > R7g,0x( — C’) 


and the fact that R?g, 0,( — C’) = 0 since fibers of g have dimension less than 2, 
shows that R'g, 0, + 0. Since R'g, Ox is supported at the point Q,h, R'g, Ox + 0. 
Examination of (20) now shows that R*h, 0, + 0, and R'h, 0, = 0. 

Z is nonsingular except at the point Q. Z is not Cohen-Macaulay at Q by 2.6.1 
[B] or Lemma 0.5 [BeSo] since g:X —> Z is a resolution of singularities, and 
R‘g, Ox is supported at Q. 


The scheme Z constructed above is obtained by blowing up an m-primary ideal 
J. Hence for some r > 0, if J = J’, By, 201" is normal and Z = proj(@, 201"). Z is 
not Cohen-Macaulay implies that @,, 201" is not S3. 
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Oblatum 19-XII-1989 


Abstract. A Gaussian integral kernel G(x, y) on R” x R” is the exponential of 
a quadratic form in x and y; the Fourier transform kernel is an example. The 
problem addressed here is to find the sharp bound of G as an operator from L?(R") 
to L‘(R") and to prove that the L?(R”) functions that saturate the bound are 
necessarily Gaussians. This is accomplished generally for 1 < p < q < 0 and also 
for p > q in some special cases. Besides greatly extending previous results in this 
area, the proof technique is also essentially different from earlier ones. A corollary 
of these results is a fully multidimensional, multilinear generalization of Young’s 
inequality. 


I. Introduction 


The classic Hausdorff-Young-Titchmarsh [T] inequality for Fourier integrals 
states that for 1 < p < 2 the Fourier transform on L?(R") is a bounded map into 
L”’(R") with a bound that is at most 1; here 1/p’ + 1/p = 1. In 1961 Babenko [BA] 
showed that when p’ is an even integer greater than 2 and n = 1 the bound is in fact 
less than 1, and he determined its value. This bound is achieved for Gaussian 
functions and Babenko states, but does not demonstrate explicitly, that Gaussians 
are the only functions with this property. Babenko’s method was to apply analytic 
function theory to the Euler-Lagrange equation associated with the maximization 
problem. 

The Fourier integral is but one example of a transform given by a Gaussian 
integral kernel G(x, y), i.e., the exponential of a quadratic plus linear form in x 
and y. In the Fourier transform case in R" the kernel is G(x, y) = exp{ — 2i(x, y)}. 
Another well known example in R” is the purely real operator 
G = exp{td + 2tx- V} on Gauss space (with measure du = exp{ — |x|?} dx) in- 
vestigated by Nelson [N1; N2] as an operator from L?(R", du) to L4(R", dy). In 
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terms of Lebesgue measure, this amounts to considering the kernel 


(1 — c*) 


from L?(R") to L4(R") for 0<c =e7'< 1. Nelson defined the operator Y by 
(Gf (x) = ) G(x, y) f(y) dy and showed that is bounded from L?(R") to L4(R") 
when p S q if and only if (q¢ — 1)c? < p — 1; he also derived the explicit value of the 
bound—which again is achieved when f is a Gaussian. This is the famous hypercon- 
tractivity theorem. [In [N1] Nelson showed that ¢ is bounded if c is small enough; 


Glimm [GL] used this fact plus the spectral gap in the generator to show that is 
a contraction on Gauss space for some still smaller c. Finally Nelson [N2] proved 
the sharp bound as stated above. In 1976 Neveu [NE] and Brascamp and Lieb 
[BL] found other proofs, and Simon [SI] found a proof for p=2 and 
q =2,4,6,8.... Recently, Carlen and Loss [CL] have used their method of 
competing symmetries to construct another proof of the hypercontractivity the- 
orem.] However, Nelson’s method seems incapable of showing that Gaussians are 
the only maximizers; the proof of this fact, as well as a completely different proof 
(using rearrangement inequalities) of the hypercontractivity theorem was given by 
Brascamp and Lieb [BL]. The method in [CL] also yields uniqueness. Nelson’s 
original proof used stochastic integrals and Gaussian processes in R" (in fact it even 
extends to infinite dimensions). Segal [S] showed how to use Minkowski’s inequal- 
ity [HLP] to reduce the R" case of Nelson’s kernel to the R’ case; he also showed 
that Y is a contraction on Gauss space for small c. The R' case was simplified by 
Gross [G] who showed the equivalence of hypercontractivity with logarithmic 
Sobolev inequalities and built up one-dimensional Gauss measure from two-point 
measures via the central limit theorem. See the survey by Davies et al. [DGS]. 

In his important 1975 paper, Beckner [B1; B2] used the Nelson-Gross machin- 
ery and the Hermite semigroup to settle the question raised by Babenko. By using 
the tensor product structure of Fourier transforms and an application of Min- 
kowski’s inequality related to, but distinct from, Segal’s [S], he reduced the R” case 
to the R' case. He also showed that for all 1 < p < 2 the sharp constant in the 
Hausdorff-Young-Titchmarsh inequality is given by Gaussian functions—as 
found by Babenko. However, this method also leaves open the question of whether 
Gaussian functions are the only maximizers. 

Since then the Nelson-Gross-Beckner method has been extended to other 
complex (as distinct from purely real or purely imaginary) Gaussian kernels in R"” 
(i.e., the complex Mehler kernel) [C; E; J; W]. In this paper the general problem in 
R” in the p < q case will be settled by a completely different method and, moreover, 
the maximizers will be shown to be Gaussian functions. Some of the p > q cases will 
be settled as well. Before discussing the earlier results in detail it is necessary to 
define the problem more completely. 


1 1 ly — cx|? 
G(x, y) = exp sa al a oer °4 


The most general Gaussian kernel on R” x R" is 


G(x, y) = exp — (x, Ax) — (y, By) — 2(x, Dy) + 2( 1 (*)) (1.1) 


and its action on complex valued, measurable functions f: R" > C, is formally 


given by 
(Gf)(x) = J G(x, y) f(y) dy (1.2) 
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In (1.1) A, B and D are (complex) n x n matrices with A and B being symmetric 
while L is a vector in C?". The Fourier transform corresponds to A = B = 0, L =0 
and D = il, with I denoting the identity. 


Notation. If « and f are vectors in C” then (a, 8) = )’7_, 4,8; and not )7_,, &,B;. 
Lebesgue integration over R” is denoted simply by { dx whenever the n in question 
is clear from the context. The L?(R”) norm of a measurable function f will be 
denoted by || f|,, Le. { J | f(x)|?dx}"/”. The notation 


A D 
= a =M+iN (1.3) 


will also be used, where M and N are real, symmetric 2n x 2n matrices. The sole 
condition imposed on G is that M is positive semidefinite. G is said to be 
nondegenerate if M is positive definite, while G is said to be degenerate if M has 
a zero eigenvalue. The Fourier transform kernel and Nelson’s kernel with 
(q — 1)c? =(p — 1) are examples of degenerate kernels. The operator Y should 
perhaps be written Y,, but this will not be done since the pairing of Y and G will 
always be clear from the context. 

The linear operator Y associated to G will be studied as an operator from 
L?(R") to L4(R") for 1 < p< o and 1 <q < o. (The cases p or g= 1 or © can 
also be analyzed by the methods of this paper but they will be omitted since these 
cases involve extra technical considerations.) When G is nondegenerate the defini- 
tion of Y in (1.2) makes sense (by H6lder’s inequality) but if G is degenerate then 
(1.2) is meaningless unless f is also in L?(R")7 L'(R"). Assuming that Y, when 
restricted to L?(R") (1 L'(R"), is bounded from L?(R") to L4(R") then, for any 


fe LR"), GfeL(R") is uniquely defined by taking any sequence 
f,€ L?(R") L'(R") that converges to f in L?(R") and then noting that 
Gf =lim,..,,9f, is well defined since Yf, is a Cauchy sequence in L4(R"). This 
definition is well known and is, in fact, the way that the Fourier transform is 
defined when 1 < p <2. 

Associated to G and the numbers p and q with 1 <p < © andl <q ow is 
the ratio 


_ IGS il 
Ifl, 


for f € L?(R"), f + 0 and, in case G is degenerate, fe L'(R") as well. The norm of 
G from L?(R") to L*(R") is defined to be 


Cog = SUP Ryne J) (1.5) 


R, Af) (1.4) 


in which the supremum is over the class of f’s just stated. In case 0 + f € L?(R") and 


Ci+q< © and 
WFF lq = Cpagll fll, 


(using the above definition of Yfas a limit when G is degenerate) then fis said to be 
a maximizer for G (or for G). If there is any ambiguity about the G under discussion 
(e.g., in Theorem 3.3) the notation #,..,(G,f) and C,_,,(G) will be used. 


pq pa 


Functions from R" to C of the form 


g(x) = wexp{ — (x, Jx) + (I, x)} 
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with 0 + » € C,/ e C" and J a symmetric n x n matrix with Re(J) positive definite 
will be called Gaussian functions. In case L = 0 in (1.1) or / = 0 in (1.6) then G (resp. 
g) will be called a centered Gaussian kernel (resp. function). If A, B, D and L in (1.1) 
are real then G is said to be a real Gaussian kernel. Likewise, if J and / (but not 
necessarily yz) in (1.6) are real then g is said to be a real Gaussian function. 

A preliminary simplification of G can be made. Without loss of generality it 
can be assumed that A and B are real matrices because the imaginary part of B can 
be absorbed into fin (1.4) without changing || f ||,. The imaginary part of A can be 
omitted without changing ||9f||,. For the same reason the vector L can be 
assumed to be real. Furthermore, when G is nondegenerate then we can also set 
L (which is now real) equal to zero. The reason is simply that the affine change of 


variables (*)-(*) — V, with V being the unique solution of the equation 


y 

MV = Lin R”’, eliminates the real linear term from (1.1) and merely changes C,_, 
into C,_,,exp{(L, V)}. When G is degenerate, L can also be eliminated in the same 
way provided MV = L has a solution. Because Rank(M) < 2n in the degenerate 
case, such a solution conceivably might not exist, but it turns out that a solution 
does indeed exist whenever Y is bounded. This is the content of Lemma 2.2 below. 
Therefore, without loss of generality, the only G’s that need to be studied are those for 
which 


(i) A and B are real, symmetric n x n matrices, 
(ii) L = 0, ie., G is centered. 


These assumptions will be made in the theorems in this paper. 

On the other hand, suppose that the supremum of &,_,,( f) in (1.5) is taken over 
Gaussian functions only (which are automatically in L?(R") for every p). Then, 
according to Lemma 2.3 below, only centered Gaussian functions need be con- 
sidered in (1.5). This is a considerable simplification that is not altogether obvious 
and it is important in the application of Theorem 4.1 which states that this 
restricted supremum is all that need be considered. 

The results of this paper can be summarized as follows. Three.cases are treated. 
With the assumptions (i) and (ii) above, 


(A) D is real and 1 <p< o andl<q< 

(B) D is imaginary and either | < p< 2 and 1 <q< o orelse 1 < p< o, and 
25q< 0. 

(C) D is complex and 1 <p<q< o. 


If G is nondegenerate then #,.,, has exactly one maximizer and it is a centered 
Gaussian function. These are Theorems 3.2, 3.3 and 3.4. 
If G is degenerate then in all cases 


C+, = sup #,..,(9) (1.7) 
9g 


where the supremum is over centered Gaussian functions. This is Theorem 4.1. 
Furthermore, if the supremum in (1.7) is achieved for some Gaussian function then, 
when p <q, every maximizer is a Gaussian function—as Theorems 4.5 shows. 
Theorem 4.3 gives a sufficient condition for the achievement of the maximum in 
(1.7) in the degenerate case; in Case (A) it is necessary as well. Thus, Case (A) is 
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settled completely: in the degenerate case with p 2 q there is no maximizer of any 
kind, while if p < q all maximizers are Gaussian functions. 

In general, the question of the existence of a maximizer in the degenerate case is 
a subtle one. For the Fourier transform (which is both Case (B) and (C)), every 
function in L?(R") is a maximizer when p = q = 2; on the other hand the seemingly 
harmless modification of the Fourier transform in 4.2(5) below is bounded but has 
no maximizer of any kind when q = p’ = 2. When q = p’ > 2 the Fourier transform 
on R' has a three real parameter family of maximizers, f(y) = exp{ — Jy” + ly} 
with J>0O and JeC. When p<q _ the’ convolution — kernel 
G(x, y) = exp{ — (x — y)?} on R' has a one real parameter family of maximizers, 


f(y) = exp{ — Jy? + ly} with ] e R and J = , — 1; when p = q, G is bounded but 


there is no maximizer (see 4.2 below). There does not seem to be any simple rule. In 
simple cases (which include all the standard ones in R" and all the cases in R') the 
existence of a Gaussian maximizer in (1.7) can be decided by computation. Other- 
wise, (1.7) reduces to a complicated algebraic problem and precise conditions are 
not given here. Moreover it is not even proved that the absence of a Gaussian 
maximizer in (1.7) precludes the existence of a non-Gaussian maximizer—although 
a conjecture to this effect is made in 4.4. 

All these results extend to Gaussian kernels on R™ x R", in which A is m x m, 
Bisn x n, Dism x nand L € C™*". The proof is given in Sect. V. This generaliz- 
ation, while it is an easy one, does occur in applications, e.g., the entropy bound for 
coherent states in [L1]. 

Multilinear Gaussian forms are discussed in Sect. VI and it is proved there that 
the methods and results of Sects. II-V carry through for real forms. As an 
application of the real multilinear result in Sect. 6.1, the fully multidimensional 
Young inequality for K functions (which was left unresolved in [BL], p. 162) is 
proved in 6.2. The method of proof is, of course, quite different from that in [BL]; 
there, rearrangement inequalities were used and they were not flexible enough to 
encompass the fully multidimensional case. 

The relationship of the results of this paper to earlier results on Gaussian 
kernels (beyond [BA; N1; N2; B1; B2]) can be summarized as follows. In 1976 
Brascamp and Lieb [BL] found the norm for Case (A) in R" (Theorem 7) and 
proved that Gaussian functions are the unique maximizer in R! in the degenerate 
case (Theorem 13); this latter proof easily extends to R” and to the nondegenerate 
case. In fact, by a simple change of variables (see the proof of Theorem 4.3 below) 
the R” Case (A) reduces to a simple tensor product of R' kernels. In 1979 Coifman 
et al. [C] used Beckner’s result and an interpolation technique to deduce the norm 
for the complex Mehler kernel in R' for q = p’ = 2 (which is in Case (C)). In the 
same year Weissler [W] extended Nelson’s and Beckner’s results to the complex 
Mehler kernel in R! with the exception of 2 < p< q <3 and 3<p<q<2.In 
1988, Epperson [E] found the norm for the following nondegenerate cases in R': 
Case (C), Case (B), the case p = 2 = q. He also found the norm for certain R' cases 
q <p <2 and 2 <q <p with sufficiently nondegenerate kernels (Theorem 2.10), 
and for the R! degenerate Case (C) if A > 0 and B > 0 (corresponding to Theorem 
4.3 here). 

The only complex cases in R” that were known prior to Epperson’s work were 
the simple tensor products of R' kernels; these could be analyzed for p < q via 
Minkowski’s inequality, as shown by Beckner [B1; B2]. Epperson was able 





184 E.H. Lieb 


to handle the nondegenerate Case (C) for which there is an n x n complex sym- 


1 
metric matrix W with ||W|| <1 such that A= W(I— batt ri 


1 
B=(I — W?)"!- 7 I and D = W(I — W?)~'. Here, I is the identity matrix. 


It will be seen from the above summary that all the previous cases, except for 
Epperson’s R' cases of p = 2 = q and the special q < p < 2 and 2 < q < p cases, 
are covered in the cases (A), (B) and (C) treated in this paper. Moreover cases (A), 
(B) and (C) are resolved here in full R” generality (i.c., not only for simple n-fold 
tensor products of R' kernels). The main methodological point of this paper, 
however, is that all the previous results, except for [BL] and [BA], ultimately rely 
on the Nelson-Gross machinery which, while it is natural in its original context of 
quantum field theory and Gauss measures, is conceptually complicated in the 
context of general Gaussian kernels with Lebesgue measure. The two settings 
(Gauss measure and Lebesgue measure) for Gaussian kernels are mathematically 
equivalent, however, and the choice is a matter of taste. Lebesgue measure is used 
in this paper because it is felt that it is more natural to retain translation invariance 
(e.g., in the Fourier transform). Prior to Epperson’s work all results in the field, 
except for [BL] and [BA] came from translating Gauss measure bounds for 
products of complex R' Mehler kernels into R" results via Beckner’s Minkowski 
lemma. The proofs here use only Minkowski’s inequality and simple facts about 
analytic functions (which appear to be unrelated to Babenko’s use of analyti- 
city—the Euler-Lagrange equation is not used). 

Basically there is one idea that runs through Theorems 3.1, 3.3 and 4.5, 
although the technicalities are different in each. The main idea is to study 9 @ Y 
from L?(R2") to L4(R2") and use Minkowski’s inequality. By considering the YJ @ Y 


maximizer F(y,, y.)=f aa, f (25) where f is a maximizer for Y, it is 


possible to conclude that f must be a Gaussian. It will be noted that some of the 
proofs are long, and so it may appear at first that their structure is not really very 
simple. To a large extent the length is due to the fact that proving uniqueness raises 
technical considerations that would be absent if only inequalities are proved, e.g., it 
is not sufficient here to prove the inequalities for a dense set of smooth functions. 

Apart from the extension to R” (which is handled here in a natural way) the 
main new theorem in this paper is that a maximizer must be a Gaussian, and it is 
unique in the nondegenerate case. In the degenerate case C,_,,(G) is determined by 
examining only Gaussian functions and, if a Gaussian maximizer exists, every 
maximizer is a Gaussian. This is Theorem 4.5 and it can be useful as in [L1] and 
[L2]. Except for the real case [BL], it was previously known that Gaussian 
functions were among the maximizers. The one exception to this rule was pointed 
out by Beckner (private communication) for the Fourier transform from L?(R") to 
L”’(R") with the restriction p’ > 4. His proof that a maximizer must be a Gaussian 
function in this case uses a result in [BL]; the proof is 


~~ a A 
IF 2 eS F ll, 2 (CH SHS lle = BCH UCSY? lhe = MCPS Md 


with r’ = p’/2 = 2, with (C?)" being the sharp Beckner (or Babenko) constant for 
the Fourier transform (denoted by ~~ ), and with y, being the sharp constant in 
Young’s convolution inequality which was derived simultaneously in [B1, B2] and 
in [BL]. A Gaussian function f(y) = exp{ — Jy? + ly} with J > 0 and I € C gives 
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equality above. However, [BL] (Theorem 13) proved that such functions are the 
only ones that give equality in Young’s inequality. 

It is a pleasure to acknowledge my debt to Eric Carlen. He helped to stimulate 
my interest in this problem and to understand the literature in the field. He also 
critically examined the work as it took shape. Thanks are also due to the Institute 
for Advanced Study for its hospitality during part of this work, and to Michael 
Loss for valuable discussions. 


II. Some basic properties of Gaussians 


2.1. Lemma (nondegenerate Gaussian kernels are compact and have maximizers). 
Let G be a centered, nondegenerate Gaussian kernel in R” x R" as in (1.1) with M in 
(1.3) positive definite and L = 0. Let 1 < p< «© and1<q< o. Then @ in (1.2) is 
a compact operator from L?(R") to L*(R") and there is at least one maximizer 
fe LR" (i.e, RyA f) = Cy+4)- 

Every such maximizer f: R" + C, has the following three properties, in which 
a and B are positive constants that depend on G, p and q but not on f. 


(a) There is an entire analytic function of order at most 2, m:C"-—C, such that 
f(x) = |m(x)|?' m(x)~! for x € R". Here 1/p + 1/p' = 1. Moreover, for z € C", 


|m(z)| S «|| fb * exp{BlzI7} . 


(b) The function | f |?"~» from R" to R has an extension to an entire analytic function 
from C" to C whose order is at most 2. If g: C" + C is this extension then for 
zeC’ 


\g(z)| S all fZ°~Y exp {Blz|?} . 


(c) For x € R” 


|f(x)| S all fll,exp{ — B(x, x)} . 

Finally, iff, € L?(R") for j = 1, 2, 3, . . .is an L? bounded maximizing sequence for 
G(i.e., B,.4( fj) C,+,) then there is a function fe L?(R") and a subsequence 
j(1), j(2), . . . such that fq) > f strongly in L?(R") as k > oo. Iff + 0 (i.e, if || fl, 79 
as j > 0) then f is a maximizer. 


Proof. For any f € L?(R"), Hélders inequality can be used to deduce 
(Ff)(x)| S THIS Il, (1) 


with T(x) = || G(x, -)||,-. Simple computation shows that there are positive numbers 
y and 6 depending only on G and p such that | 7(x)| S y exp{ — 6(x, x)}. The fact 
that G is nondegenerate is crucial for this result. The fact that T €¢ L'(R") 0 L®(R") 
shows that Y is bounded from L?(R") to L*(R”). Now suppose that f; « L?(R”) is 
a sequence that converges weakly in L?(R") to some f € L?(R") as j > 00. Since, for 
each x € R", G(x,:) is in L?(R"), it follows that (Y f(x) > (Y f(x) as j > 00 for 
each x € R". It can be assumed that the f, and f satisfy || f; ||, and || f ||, < C for some 
C > 0 and hence, from (1), the functions Yf; and Yf are bounded pointwise by the 
function CT. Since T € L*(R"), || Gf, — Gf ||, + 0 by dominated convergence. Thus 
G takes weakly convergent sequences in L?(R") into strongly convergent sequences 
in L4(R"), and so Y is compact. 
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Now let f; be a bounded maximizing sequence, i.c., 2,.,(f;) + C,.q as j— ©. 
We can assume || f; ||, = 1 for each j. By the Banach-Alaoglu theorem, there is an 
fe L?(R") and a subsequence j(1), j(2), . . . such that f; > f weakly in L?(R"). As is 
well known, || f ||, < 1. Then, by the strong convergence proved above 


Cog = lim 9 fia lla = IFf,SC, I fl,>s 


This implies that || f||,= 1 and that f is a maximizer. Moreover, the fact that 
\| fp = 1 implies (by the uniform convexity of the L’? norm) that fi, converges to 
f strongly in L?(R"). Thus, the first and last assertions of the lemma have been 
proved. 

It remains to prove that a maximizer f satisfies conditions (a), (b) and (c) anti it 
suffices to assume that || f ||, = 1. There is a function h € L*(R") such that || h\lq 
1 and C,..,= ||9f\l, = {A(x Gf)\(x)dx. Let 


m(y) = § G(x, y)h(x)dx = e~ BY f e— 42-26 PM h(x) dx (2) 


so that, as in the proof of (1) above, |m(y)| < W(y) = wexp{ — v(y, y)} for suitable 
positive numbers yp and v which depend only G and q. Hdlder’s inequality implies 
that the function (x, y)+ h(x)G(x, y) f(y) is in L'(R" x R"), and Fubini’s theorem 
then implies that || 9 f ||, = J m(y) f(y)dy. If m(y) = |m(y)| exp {10(y)}, the optimum 
choice for f is f(y) = |m( y)|/\\m||,]? —* exp{ — i0(y)}, for otherwise Z,_.,(f) can 
be increased. 

The function m:R" — C has an extension to an entire analytic function on C" of 
order at most 2. This can be seen easily from the representation (2) above and 
Holder’s inequality; if y; = u; + iv; forj=1,...,nand D=E + iH with u,,v,,E 
and H real then 


|m(y)| S exp{(v, Bv) —(u, Bu)} [J exp{—q'(x, Ax) — 2q'(x, Eu) + 2q'(x, Hv)}dx]'/" 
= (const.)exp {(v, Bv) — (u, Bu) + (Eu — Hv, A~'(Eu — Hv))} . 


Thus |m/(y)| < (const.)exp { (const.)[(u, u) + (v, v)]} which implies that the order of 
m is at most 2. | This establishes conclusion (a). Since m is entire, the function 
yt++m*(y) = m(y) (with the bar denoting complex conjugate) is also entire, and 
hence N(y) = m(y)m*(y) is also entire with order at most 2 and with a pointwise 
bound that is independent of Sf However, when y € R"(i.e., v; = 0 for all j) then 
N(y) = |m(y)|?. Conclusion (b) is then an immediate consequence of the relation 
between f and m which implies that for y € R’, | f(y)|??~» = ||m||>-?|m(y)|? = 
|| m|| 5-7 N(y); thus | f|?”~" has an analytic extension of order at most 2, namely 
\|m||,7N. It only has to be shown that ||m||>-? p is universally bounded, but this 
follows from the relation C,., = || Gf ||, = J mf = ||m||,.. 
Conclusion (c) follows from the fact that silos yéR" then |f(y)|= 
Cim(y)|/||m ||]? ~* S Wy) mI”. O 


The next two lemmas validate the assertion in Sect. I that linear terms can be 
eliminated from Gaussians. 


2.2. Lemma (elimination of linear terms from Gaussian kernels). Let G be the 
(degenerate or nondegenerate) Gaussian kernel given in (1.1) with positive definite or 
semidefinite real quadratic form M in (1.3) and with real linear term L € R". Let Gy 
denote the Gaussian kernel with no linear term, which is obtained from G by setting 
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L=0, i.e., Go(x, y) = G(x, ypexpy - 2(t, (*))} Let l1Spso andisqso. 
Then the following conditions are equivalent. 


(i) Go is bounded from L?(R") to L*(R") and the equation MV = L has a solution 
VeR*. 
(ii) G is bounded from L?(R") to L4(R"). 
In case these conditions are both satisfied the relation between the norms is 
C,+4(G) = C,.,(Go)exp{(L, V)} . 
The number (L, V) is uniquely defined even if the vector V is not unique. Y has 
a unique maximizer if and only if Y has one. 
Proof. (i) = (ii). This was explained in Sect. I. Simply change variables; writing 


V= f plat x ++ and y+ y +b. Then 


GG exp (L V)- ai( (*) wv) ~i(V, wy} 


The imaginary terms above do not affect the norm. Since M is Hermitian L must be 
orthogonal to # = kernel of M c R”", while any two solutions V, and V, differ by 
an element of #. Thus, (L, V) is unique. This change of variables also shows that 
G has a unique maximizer if and only if Y, has one. 

(ii) > (i). Suppose that MV = L has no solution. Then, since M is Hermitean, 


L is not orthogonal to 4 and thus there is a vector W = (*) € # such that the 


number P =(W,L) is positive. Make the change of variables x >x +s and 
y—>y+t. Then, since MW = 0, G becomes 


G(x, y) = G(x, resp ~ i(W, NW) — ai((*), nw) + 2P} 


The change of variables is an isometry so the norm of G is the same as the norm of 
@G and, since the imaginary terms are irrelevant, we have C,_.,(G) = C,_.,(G)= 
e?PC,_,,(G). This is a contradiction since C,,,(G) + 0. Thus MV = L has a solu- 
tion and the same change of variables can be made as before to derive the relation 


between the norms of Y and 9). O 


2.3. Lemma (elimination of linear terms from maximizers). Let G be a centered 
Gaussian kernel (degenerate or nondegenerate) and let 1 < p< «© and1<q< ©. 
Assume 9: L?(R") > L4(R") is bounded (which is automatically true in the non- 
degenerate case). If g(x) = exp{ — (x, Jx) + (I, x)} is a Gaussian function that maxi- 
mizes @,.,,(g) among all Gaussian functions then g(x) = exp{ — (x, Jx)} is also 
a maximizer. Moreover, if #,.,(g) does not have a maximizer among Gaussian 
functions (which can happen only if G is degenerate) then the supremum of #,..,(9) 
over Gaussian functions equals the supremum over centered Gaussian functions. 
Finally, if G is nondegenerate then g = go, i.e., | = 0, and therefore g is centered. 


Proof. Consider the functions g,(x) = exp{ — (x, Jx) + A(l, x)} with / a real para- 
meter. Clearly g, € L?(R") for all 4 and, by a well known property of Gaussian 
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integrals, 


I9allp=lgollpe™ and Ggall,=Faoole” 
for some real constants « and f. There are three cases to be considered: 


(i) a > B. By setting 2 = 0, B+ is increased, i.¢., 2, .4(9o) > B 
that g is not a maximizer—which is a contradiction. 

(ii) « < B. By letting 4 tend to infinity we conclude that # 
unbounded—which is a contradiction. 

(ii) « = B. In this case g, is a maximizer for every A and hence gp is a maximizer, as 
claimed. 


p~q(g)- This means 


pq (and hence also ¥) is 


These considerations prove all but the last sentence of the lemma. 

If G is nondegenerate it is possible to go further. Consider the following 
sequence of functions with 4 = j, namely h; = Z ;9; for j = 1,2, 3,..., where the 
numbers Z; are chosen so that ||h;||, = 1 for each j. This is a bounded maximizing 
sequence and, by a trivial modification of the last part of Lemma 2.1 (using the fact 
that a nonzero L?(R") weak limit of Gaussian functions is a Gaussian function), 
there is a nonzero Gaussian function h € L?(R") and a subsequence j(1), j(2), ... 
such that hj, > h strongly in L?(R") as k > oo. If] + 0, however, it is easy to check 
that h; +0 weakly in L?(R") as j— 00. This contradicts the supposed strong 
convergence to a nonzero function. 0 


III. Nondegenerate gaussian kernels 


A main ingredient in the following theorems is Minkowski’s inequality for inte- 
grals. It was exploited by Beckner [B1; B2] to prove that the sharp bound for the 
tensor product of two operators (e.g., Fourier transforms) is often the product of 
the individual bounds. In particular, the bound for the Fourier transform from 
L?(R") to L?(R") is (C2)", where Cf is the sharp constant for R’. A proof of 
Minkowski’s inequality can be found in [HLP]. Of crucial importance here is the 
sharp form in which the necessary and sufficient condition for equality is specified; 
this condition was not used before to analyze Gaussian kernels. 


3.1. Lemma (Minkowski’s inequality). Let f:R" x R"-—[0, 0] be Lebesgue 
measurable and let 1 < r < 0. Suppose that the measurable function M, defined for 
almost every x € R" by 


M(x) = f f(x, y!'dy, 
us 


is finite for almost every x and that M'"" € L'(R"). Then the measurable function 


N(y) = J f(x, y)dx 
e 


is finite for almost every y € R™ and 


jinb sf mn, 





Gaussian kernels have only Gaussian maximizers 189 


Furthermore, if r> 1 and if there is equality in (*) then there are nonnegative, 
measurable functions A € L'(R") and B € L’(R™) such that 


F(x, y) = A(x) B(y) 
for almost every (x, y) € R" x R”. 


Remark. This lemma extends to an arbitrary pair of measure spaces (X, y) and (Y, v) 
in place of (R", dx) and (R”, dy) when yp and v are sigma finite. 

As a first application of Minkowski’s inequality the uniqueness of maximizers 
for real, nondegenerate Gaussian kernels for all p and q will be proved. This is Case 
(A) of Section I. It is to be noted that the order of integration in Theorem 3.2 is as in 
[S] and is opposite to that of Theorem 3.4 and opposite to the order in Beckner’s 
lemma. Analyticity considerations play only a subsidiary role in Theorem 3.2 and 
can be bypassed if desired, but they are important later. Theorem 3.2 was already 
essentially contained in [BL] Theorems 7 and 13. The following proof is offered 
because (i) it is different from the [BL] approach and (ii) it illustrates the techniques 
of the present paper. 


3.2. Theorem (unique Gaussian maximizer for all p and q in the real nondegener- 
ate case). Let G be a real, nondegenerate, centered Gaussian kernel, i.e., the matrix 
N in (1.3) is zero. Let 1<p< o and 1<q< o. Then @ has exactly one 
maximizer, f, (up to a multiplicative constant) from L?(R") to L*(R") and f is a real, 
centered Gaussian, i.e., f(x) = exp{ — (x, Jx)} with J being a real, positive definite 
matrix. 


Proof. Consider the linear operator 9?) =  @ Y: L?(R2") — L*(R") given by the 


Gaussian kernel G'?)((x,, x2), (V1, ¥2)) = G(x1, ¥,)G(x2, y2) with x,, x, y, and y, 
in R". The first goal is to prove that C,_.,(G) = C,.,(G)’. If F e L?(R*") then 
(V1 V2) > G(x, X2), (V1, ¥2)) F()1, ¥2) is in L*(R*") for every (x,,x,) because 
G) is nondegenerate. Fubini’s theorem and Minkowski’s inequality yield 


IGF 2 =f {IP Cf Glx,, y,G(x2, v2) F (V1, V2) dy, )dy2|* dx, } dx, (1) 
< J {JL Glx2, y2)/K(%1, y2)ldy2]*dx, }dx, 

(with K(x,, ¥2) = J G(x1, ¥1)F 1, ¥2)dy1) (2) 

<f {J LJ Glx2, yo) K(x, y2)[*dx, ]!* dy, }4dx, 3) 

S(C,.4(G) J {J Glx2, y)EJ IF, ya)? dy 1"? dy, }4¥ dx, (4) 

= (C,.q(G))*{ i} IF (v1, y2)I?dy, dy. }””. (5) 


(Notes: (2) > (3) is Minkowski’s inequality. (3) > (4) uses C,_.,(G) 2 #,..,(F (-, ¥2)) 
for each y,. (4) (5) uses C,+4(G) = B,~4((f| Fv.) dy,)"/”). The fact that 
G(x, y) 2 0 is crucial. Here the x, integration was done before the x, integration; 
in Theorem 3.4 the x, integration will be done first.) Inequalities (1}{5) establish 
that C,.(G™)<C,_,(G)’. Clearly, by considering F’s of the product 
form F (y,, y2) = h(y,)h(y,), the reverse inequality is obtained, and so the goal is 
reached. 

Suppose now that F: R*" > C is a maximizer for G’. Since G) is nondegener- 
ate, it has a maximizer by Lemma 2.1. Since G(x, y) > 0 for all x and y, it is clear 
that F = A|F| and |A| = 1, for otherwise replacing F by |F| will increase the 
quotient #,.., for G. It can be assumed henceforth that F 20. Since F is 
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a maximizer all the inequalities in (1)}-(5) must be equalities. Equality of (2) and (3) 
implies, by Lemma 3.1, that fer almost every x, there are measurable functions A,,, 
and B,,:R" — (0, 00) such that 


G(x, Y2)K(x,, ¥2) = A,,,(X,)B,,(¥2) (6) 


for almost every x, and y,. Since G > 0, this equation can be divided by G(x, y,) 
to obtain K(x,, y2) = A,,(x,)E,,(y2) with E,,(y) = B,,(y)/G(x2, y). However, 
K(x,, y2) is independent of x, and therefore if any particular value of x, is chosen 
for which (6) holds for almost every x, and y,, and if the functions A and 
E:R"-—[0, 00) are defined by A = A,, and E = E,, for this value of x,, then 


K(x, Y2) = A(x, )E(V2) 
for almost every x, and y,. If this equation is multiplied by G(x,,y,) and 
integrated over y, the result is 
(GF (x, X2) = A(x1)Z(x9) 


for almost every x, and x, with Z = GE. Since G > 0, both A and Z are strictly 
positive functions. 

There is a function H € L*(R*") with ||H ||, = 1, such that ||9F ||,= 
fH:9F . In fact 


H(x,, x ) = (const.)[(G™ F )(x,, x) ]*~ ! = (const.)A(x,)?~1Z(x,)*7!. 


The point here is that H is a product function. Then, as in the proof of Lemma 2.1, 
F satisfies 


F(y,, y2) =(const.) { fj G(x, ¥;)G(X2, Y2)H(x,, X2)dx, i" * = a(y,)B(y2) (7) 


for some positive function « and B:R"— [0, 00). In brief, F must be a product 
function, and this fact is crucial for the next step. 

One example of a maximizer is F(y,, y,)=/f(y,)f(y2), where f is an 
L?(R") > L*(R") maximizer for G (whose existence is guaranteed by Lemma 2.1). 
For the reason given before about F, we can and do assume that f(x) = 0 for all 
x eR". 

A more interesting maximizer is 


F (y;, Ya) =i(2 = “)i(2 ae 2) (8) 


Here, the essential property of O(2) rotation invariance of products of centered 
Gaussians and of Lebesgue measure is being exploited. If 0 is any fixed angle and if 
x4,X2,¥i,y2 in R” are defined by x, =x, cos?—x,sin@, x,=x,sin@+ 
x, cos 0, y; = y, cos 8 — y, sin, y, = y, sin 8 + y, cos 0, the O(2) invariance of 
Lebesgue measure is that dx,dx,=dx'dx, and dy,dy, =dydy2. The O(2) 
invariance of centered Gaussian functions is that g(x,)g(x.) = g(x)g(x2), while for 
centered Gaussian kernels G(x,, y,)G(x2, y) = G(x}, y1)G(x2, y2). With the 
choice @ = 7/4, these observations lead to (8). Combining (7) and (8), 


Yi —J)2 tinal 3 9 
s( 2B )a( oe) a(y,)B(y2) . (9) 


for almost every y, and y,. 
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Equation (9) implies that f is a Gaussian. Instead of proving this in full 
generality for L?(R") functions, as is done by Carlen [CA], it is easier to simplify the 
proof here by taking the 2(p — 1) power of (9) and by taking advantage of the 
analyticity result Lemma 2.1(b). Introducing h=f?"~), y=a?-) and 
5 = B?"?~”) it is seen from (9) (by fixing y,) that y is analytic; likewise 6 is analytic. 
Thus, (9) holds for all y, and y, because when two analytic functions on C” x C" 
agree almost everywhere on R” x R” then they agree everywhere. Furthermore 
f never vanishes for real y because if f(Y) = 0 then, setting y, = y, + /2 Y, we 
would have that 0 = p(y, + “fa Y)d(y,) for all y,; this is impossible, given that 
y and 6 are analytic, unless y = 0 or 6 = 0, which contradicts the assumption that 
f # 0. Thus, the logarithms of h, y and 6 are real analytic and 


in| (22) | + in| (272) | = In[y(y,)] + In[5(y,)]. (10) 


If 0, denotes the derivative with respect to the i coordinate, and 0; with respect to 
y, and 0; with respect to y, is taken in (10), then 


(@¢,1n)(% = 2) o (jinn (% : 22) 
: J/2 J/2 


which implies that the function 0;0;Inh is a constant (call it 4(1 — p)J,;) and 


therefore In[ f(y)] = on In[h(y)] = — (y, Jy) + (l, y) for some vector /. Ac- 


cording to Lemma 2.3, / = 0 since G is centered and nondegenerate. This completes 
the proof that f must be a centered Gaussian. 

It remains to prove that fis unique (i.e., the matrix J above is unique). One way 
would be to compute #,_,,(exp{ — (x, Jx)}) for G and then deduce that there is 
only one optimum J. A very much easier route is to suppose that there are two 
maximizers f' and f? with f‘(y) = exp{ — (y, J‘y)}. Then, for the same reason as 
before (O(2) symmetry) the function 


Pty, ill 2) (242) 11 
(V1. ¥2) s( 2 f 2 (11) 


is a maximizer for Y‘”’. There are two ways in which this implies that f' = f?. The 
first is to use (7), namely F must be a product function, and to note that this 
product structure is true if and only if J' = J”. The second way is to note that since 
the F in (11) is never zero and, since (3) > (4) must be an equality, we have that the 
function y,++h,,(y,) = F(y,, y2) must be a maximizer for 9 for almost every y,. 
Although the function h,, is a Gaussian for each y,, the Gaussian will have a linear 
term for each y, + 0 unless J' = J*. However, Lemma 2.3 precludes the existence 
of such a linear term, so J' = J*. 0 





The next theorem concerns Case (B) of Sect. I. 


3.3. Theorem (unique Gaussian maximizers in the imaginary, nondegenerate case). 
Let G be a centered, nondegenerate Gaussian kernel with a real diagonal part and 
a purely imaginary off-diagonal part, i.e., 


G(x, y) = exp{ — (x, Ax) — (y, By) — 2i(x, Dy)} 
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where A, B and D are real n x n matrices and A and B are positive definite. Let 
1<ps2and1<q< o orelsel1<p< © and2<q& oO. Then, in either case, 
G has exactly one maximizer, f, (up to a multiplicative constant) from L?(R") to L4(R") 
and this f is a real, centered Gaussian, i.e., f(x) = exp{ — (x, Jx)} with J being a real, 
positive definite matrix. 


Proof. Assume at first that D is nonsingular. Since A and B are positive definite 
there are nonsingular real matrices U and V so that the change of variables x + Ux 
and y + Vy changes A and B to the identity matrix, I, that is ] = U7 AU = V" BV, 
where T denotes transpose. Then (x, Dy) = (x, Dy) with D = UTDV. The polar 
decomposition of D is D = W\DI, where W is orthogonal and |D| is positive 
definite (the assumption that D is nonsingular i is used here). Then there is an 
orthogonal matrix Y such that Y7|D|Y is diagonal and there is a real diagonal 
matrix Z such that ZY7|D| YZ = I. Now make one more change of variables: 
x—+WYZx and y>YZy so that (x, Dy) > (WYZx, W|D|YZy) =(x, y) and 
(x, x) = (x, Ix) > (WYZx, WYZx) = (x, Z?x) and (y, y) > (¥Zy, YZy) = (y, Z’y). 
These two changes of variables affect 2,.., in a trivial way (involving only p and 
q and the determinants of U,V and Z) and, most importantly, take Gaussian 
functions into Gaussian functions. In short, it can be assumed without loss of 
generality that G has the canonical form 


G(x, y) = exp{ — (x, Ax) — (y, Ay) — 2i(x, y)} , (1) 


where A is positive definite and diagonal. 

By duality C,_.,(G) = C,,.,(G") with G7(x, y) = G(y, x) = G(x, y), so it suffi- 
ces to consider only the case 1 < p<2 and 1 <q< oo. It is easily seen that 
(Gf)(x) = exp{ — (x, Ax)}h(x) where h is the Fourier transform of the function 
h(y) = exp{ — (y, Ay)} f(y). Since fe L?(R") it has a Fourier transform f, and 
Beckner’s theorem (which will also be proved here in Theorem 4.1 and 4.2(1)) states 
that || f ll S(C5)" If ||, where Beckner’s constant C? is the sharp constant for the 
p—p’ norm n of the Fourier transform in R’. By the convolution formula, h satisfies 


h(x) = wf exp{ —(x — y, A7'(x — y)} fdy, 


where p > 0 is a constant which depends only on A. Therefore (9 f)(x) = WG f(x) 
where G is the real, centered, nondegenerate Gaussian 


G(x, y) = exp{ — (x, Ax) — (x — y, A71(x — y))} . (2) 
Thus 


RAGIN Np = UR yp AGANS lp S UCy (OIF ll S UC, ro KCTS ly 


from which it follows that C,_.,(G) < u(C®)"C,,.,(G). However, equality can be 
achieved in (3) in exactly one way (up to a multiplicative constant). By Theorem 3.2 
there is exactly one choice for f that will make the first inequality in (3) into an 
equality. This f is a real, centered Gaussian, f(x) = exp{ — (x, Jx)}. Its inverse 
transform fis also a real, centered Gaussian, i.e., f(x) = (const.)exp{ — (x, J ~'x)}. 
The second inequality in (3) (Beckner’s) is an equality for any real Gaussian (in 
particular, our f), and therefore f is the unique maximizer as asserted in the 
theorem. 
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In case D is singular, a change of variables similar to the above replaces the 
canonical form (1) by 


G(x, y) = exp{ — (x, Ax) — (y, Ay) — 2i(x, Py)} 


5. Ot. : area , 
where P = ( 0 5) is a diagonal projection onto R”™ (with m < n being the rank of 


a 
D)and A = 
)an O71 
(x,,X2) with x, e R” and x, € R"-™, define g:R” > C by 


giyi)= J exp{—(y2,y2)} S01, y2)dy2 » (4) 
R= 
and G*:R™ x R™ > (0, 00) by G*(x,, y1) = exp{ — (x,, ax,) — (y,, ay) 
— 2i(x,, y,)}. Then, using Fubini’s theorem and the same analysis as before with 
a and m in place of A and n, and with G*:R” x R™ > (0, 00) as given in (2), 


Ry oh GA INF \ rem) = VR pag G*, DING ll ew S VEC yp ~g(G* CF)" Ig lle (5) 


where v is the L7(R"~”) norm of the Gaussian function exp { — (x,, x,)}. As was the 
case for (3) and the subsequent argument, equality in (5) is uniquely achieved by 
a real, centered Gaussian 


with « positive definite, m x mand diagonal. Writing x € R” as 


g(y,) = exp{ — (1, Ey,)}, (6) 


where E is a real, positive definite m x m matrix. By Hdélder’s or Minkowski’s 
inequality, it follows from (4) that 


9 lz) S N\\f |e) (7) 


where N is the L?(R"~™) norm of the Gaussian function exp{ —(y,, y,)}. Equality 
in (7) is compatible with (6) if and only if f(y) =f(y,, y2) =(const.)exp{ — y,, Ey,)— 
(p' — 1)(y2, y2)}. The reason is that equality in (7) requires that 
f(¥1, ¥2) = A(y, exp { — (p’ — 1)(y2, y2)} for almost every y, € R”. Then, comput- 
ing the integral in (4), one finds that exp{ — (y,, Ey,)} = g(y,) = (const.) A(y,). 0 


Finally, Case (C) of Sect. I will be considered. 


3.4. Theorem (unique Gaussian maximizer when p < q in the general nondegener- 
ate case). Let G be a centered, nondegenerate Gaussian kernel and let 
1<p<q< o. Then Y has exactly one maximizer, f, (up to a multiplicative 
constant) from L?(R") to L4(R") and f is a centered Gaussian function. 


Proof. As in the proof of Theorem 3.2, the key is to study the kernel G = G @G 
by means of Minkowski’s inequality. Now, however, the x, integration is done first. 
Thus, for F e L?(R"), 


WOOF 18 =f {SlJ Ger, vf Gey, Fi, vaddy,)dy2|*dx,}dx, (1) 
= J { fij G(x2, V2) K(x, y2)dy2|*dx2}dx, 

(with K(x,, y2) = J G(x, y:) FQ. ¥2)dy1) (2) 

S (C,.4(G))" J { J | K (x1, y2)/Pdy2}/"dx, (3) 

S (C,.(G))* {JCJ |K (xy, v2) dx, JP'4dy, }”” (4) 

S (C,.4(G))4 {JJ F 1, yo)lPdy, dy2}t”” . (5) 
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(Notes: (2) (3) uses C,_,,(G) 2 #,.,(K(x,,°)) for each x,. (3)>(4) is Min- 
kowski’s inequality for the exponent r = q/p 2 1 and the function |K(x,, y,)|?. 
(4) — (5) uses C,_,,(G) = #,.,(F (:, y2)) for each y,.) This inequality (along with 
consideration of F’s of the form F(y,, y,) = h(y,)h(y2)) shows that 


C,.(G) = C,.(G) - 


Suppose now that F:R2"->C is a maximizer for 9). Since G?) and G are 
nondegenerate, maximizers exist for each of them by Lemma 2.1. Then all the 
inequalities in (1}{5) must be equalities. In particular, inequality (4) — (5) implies 
that the function y,++F(y,, y,) must either be the zero function or it must be 
a maximizer for Y for almost every y, € R”. (It is well known that this function is in 
L?(R") for almost every y,.) As in the proof of Theorem 3.2, the O(2) invariance of 
G) implies that the function given by 


F(y;, y2) -s(2 i “)r(% v7 ’) (6) 


is a maximizer for Y?) when fis a maximizer for Y, as will henceforth be assumed. 
Thus, for almost every z in R”, the function g_,(y) = F (y, z) is in L?(R") and either (a) 
it is a maximizer for Y or (b) g, is the zero function. The second possibility (b) can be 
excluded by Lemma 2.1 (b). If g, = 0 then, from (6), f(w) = 0 for all w in some set 
Ac R" of positive Lebesgue measure. But | f |?) is analytic and this is imposs- 
ible unless f = 0. Thus it can be assumed that g, is indeed a maximizer for almost 
every Z, i.e., g, + 0. 

In fact g, is an L?(R") maximizer for every z € R". To prove this assertion, fix 
zand let z,,2,,.... be any sequence in R" such that z; > z as j > oo and such that 
g., is an L?(R") maximizer for each j. Such a sequence exists because g, is 
a maximizer for z’s in a dense set. Define h,(y) = Z,g_,(y) where Z; is chosen so that 
\|h; ||, = 1 for each j. By Lemma 2.1, there is a subsequence (still denoted by h;) and 
a maximizing function h € L?(R") such that h; > h strongly as j > oo. By passing to 
a further subsequence this convergence can also be assumed to be pointwise almost 
everywhere. However, translation is a continuous operation in L?(R") and thus, by 


passing to a further subsequence, f((y + z )//2) converges pointwise to 
Ky + 2)//2) for almost every y. Likewise, by passing to a further subsequence, 


Ky - z))/./2) converges pointwise to f((y — 2)/./2) for almost every y. It follows 
then that the maximizer h satisfies 


h(y) =A? : “)s(% . ‘) lim Z, 
BV\ fa) me 
for almost every y. Therefore lim;..,,.Z; exists and g, is a maximizer for every 
ze R". 
Our first application of this result will be the proof that there is a Gaussian 


maximizer. Take z = 0 so that f(y) =f (y//2)? is a maximizer. Then apply the 
same conclusion to f) so that f(y) = f(y/2)* is also a maximizer. Repeating this 
indefinitely, the sequence of L?(R") functions given by 


9) = ns(Z) (7) 


p~4q 
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is a sequence of maximizers for j = 2, 4, 8, 16, . . .. The number Nis chosen in each 
case so that ||g;||, = 1. Using Lemma 2.1 again we infer the existence of a subsequ- 
ence (still denoted by j) and a maximizer g such that g, > g strongly in L?(R") and 
pointwise almost everywhere. Our goal will be to prove that g is a Gaussian. This 
can be inferred from the central limit theorem, but the following argument is more 
direct and will be needed later for the proof that every maximizer is a Gaussian. 

The first step is to prove that f(0)+0. Recall from Lemma 2.1(b) that 
R=|f|?"~ is analytic. Likewise S = |g|?~ is also analytic and 


J 
S(y) = lim N2-YR (2,) (8) 
iv© J 
for almost all y € R”. Since S,(y) = Nj?~ YR(y/./j¥ is the 2(p — 1) power of the 
modulus of a maximizer with unit L?(R") norm (namely g;), Lemma 2.1(b) states 
that the analytic extension of S; is uniformly bounded on compact subsets of C’”. 
The almost everywhere convergence in (8) then implies (by Vitali’s theorem) that (8) 
holds for all y € C” and that all partial derivatives with respect to y of the sequence 
of functions S; also converge as j— oo to the corresponding derivatives of S. 
However, it is easily seen by Leibniz’s rule that if R(O) = 0 then every derivative of 
S, at y = 0 converges to zero as j > 00. This is impossible unless S(y) vanishes 
identically, which contradicts the fact that ||g||, = 1. 

The second step is to prove that g is a Gaussian. By Lemma 2.1(a), for y € R", 
f(y) = |m(y) |? /m(y), where m: C" = C is entire analytic. Since f(0) + 0, also m(0) + 0 
and hence there is a neighborhood U of 0 € C" on which fhas an analytic extension 
and on which fis never zero. [Reason: m,(y) = Re(m(y)) can be written as a Taylor 
series for y € R", and so can m,(y) = Im(m(y)). Consequently m, and m, extend to 
entire functions. Then (mj+m3)?’? is analytic on  U _ since 
m, (0)? + m,(0)? = |m(0)|? + 0.] Therefore f has a logarithm, H, which is analytic 
on U, ie, f(y) =f(0)exp{H(y)}. The function H can be written as 
H(y) = (V, y) — (y, Jy) + O(y?) for some V € C" and J a symmetric matrix. For 


each y € R”, the point y/Jj lies in U for all sufficiently iarge j and therefore, by (7), 
g(y) = lim Nf exp{ /i(V, y) — (y, Jy) + OW37-1?)} 
rs 


for almost every y € R”. The factor exp{O(y?j~ '/”)} converges to 1 as j > oo and 
thus 


a(y) = exp{ — (y, Jy)} lim N, fOY exp{ Vi(V, y)} 


Clearly this last limit can exist for almost every y if and only if V = 0 and N, f(y 
has a finite limit (which cannot be zero since ||g||,, = 1). This proves that g must be 
a Gaussian as claimed (and hence Re(J ) is positive definite) but we also note that 
the argument also proves the following three statements: Whenever fis a maximizer 
then (i) f is analytic in some complex neighborhood of 0; (ii) f(0) +0; 
(iii) (6 f/éy')(0) = 0, for i=1,...,n. 

The second assertion of the theorem is that every other maximizer, f, is 
proportional to the one just found, namely g(y) = exp{ — (y, Jy))}. Instead of (6) 


take 
F( > SB io ee Vit y2 
YI =9 Ja f /2 
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which is obviously also a maximizer for Y”). By the same reasoning as before, F has 
the property that y++k,(y) = F(y, z) is a maximizer for each fixed z € R". By the 
three statements just made above, we conclude that k, is analytic near 0, 


k,(0)+0 and (0k,/dy'))=0. 


This is equivalent to the statement that for every z € R", fis analytic near z/./2, 


f(z/,/2) + 0 and 
r,) 0 i 5) =f — Jzi. (=) _ 


which shows that f= g. 0 


IV. Degenerate Gaussian kernels 


In the three cases (A), (B) and (C) of Sect. I, which correspond to Theorems 3.2, 3.3 
and 3.4, every nondegenerate Gaussian kernel has a unique maximizer which is 
a Gaussian function. By taking suitable limits the following formula 4.1(#*), which is 
one of the main results of this paper, can be deduced for the L?(R") to L4(R") norm 
of degenerate kernels. This formula is, of course, trivially true in the nondegenerate 
case. 


4.1. Theorem (the sharp bound for degenerate kernels). Let G be a centered 
Gaussian kernel as in (1.1) with L =0 and let p and q satisfy the appropriate 
conditions given in (A), (B) or (C) of Sect. I, according to the properties of G. Then G is 
bounded from L?(R") to L*(R") if and only if the following supremum is finite, in which 
case the supremum is equal to C,,,,. 


sup 2, ..,(9) = C,.,; (*) 
g 


where the supremum is taken over all centered Gaussian functions, and in Cases (A) 
and (B) they can be restricted to be real. 


Proof. For each e>0O let h,(x)=exp{—e(x,x)} and define G,(x, y)= 
G(x, y)h,(x)h,(y), which is nondegenerate. Correspondingly, there is the linear 
operator Y,. For each f € L?(R") 


RyadGe INF p= WA GAL Mg < VAAL lg S Cp- (IAL Il, 
< Cy. G)IS lly - (1) 
This proves that C,_.,(G,) < C,..,(G). 
On the other hand, assuming that C,_.,(G) < 00, for each 6 > 0 there is an 
f° € L(R") with || f°||, = 1 such that ||9f"||, > C,.,(G) — 6. Then 
Cy+qG,) = Rpg Gof?) = GIL iq = WAGASL lq - (2) 


Ase 0,h,f* +f? strongly in L?(R"), so Y(h, f°) > Y( f°) strongly in L4(R"). This 
implies that h,Y(h, f°) + G(f*) strongly in L4(R") as well, and thus, from (2), 
lim inf,_,9C,+4(G,) 2 C,..,(G) — 6. Since 6 is arbitrary, and in view of (1), 


lim C,..9(G,) = CpG) - (3) 


pq 


A similar argument shows that (3) holds even if C,_,.(G) = oo. 


p~4q 
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Now let g, denote the maximizer for G,, which is a centered Gaussian function. 
Assume ||g, ||, = 1. Then 


A.Ge ll pCp+q(G) 2 l1h.Gell pRp+q(G, hg.) = | F(h.g.) liq 
2 ||hF(h.Ge) liq = 1 FelGe) liq = Cp+qlG.) - (4) 


Assuming Y to be bounded, (4) together with (3) and the fact that 
h.gellp S \gell, = 1 implies that ||h,g,||,— 1 as e+ 0. Then 


C,.,(G) = lim C,..,(G,) < lim &, .,(G, h,g,) $ C)+qG) - (5) 
270 270 


This proves the theorem in the bounded case since h,g, is a Gaussian function 
(which is real in Cases (A) and (B)). 


In case Y is unbounded, (4) and (5) imply that 


co = lim Cy .4(G,) $ lim lh,9el], Rp -4(Gs hg.) Slim &, .(G, hg.) 


which proves the theorem since h,g, is a Gaussian function. 0 


4.2. Remarks and examples. Theorem 4.1(*) is a formula for the L?(R”) > L*(R") 
norm of Y. The same formula is, of course, also valid for nondegenerate kernels, but 
in that case we are assured that there is precisely one g that achieves the supremum. 
In the degenerate case a maximizer may not exist—even if Y is bounded—as the 
examples below show. In any event, the evaluation of this formula is, in general, 
a difficult nonlinear algebraic exercise, although it is simple in many applications. 

For example, when G(x, y) = exp{2i(x, y)} (the Fourier transform kernel), it is 
easy to deduce from 4.1(*) that G is bounded if and only if q = p’ = 2, in which case 
a Gaussian function is a maximizer if and only if it has the form 


g(x) = wexp{ — (x, Jx) + (I, x)} 


with J positive definite, real and symmetric and | € C. Both J and / are arbitrary. 
This g is not necessarily centered even though G is. In the degenerate case it is not 
asserted that every maximizer must be centered when G is centered. The sharp 
constant is then C,_.,, = (C})" with 


Cp = nile phl2r(p') 10 (1) 


[Note: The Fourier transform is an example of both Cases (B) and (C). While the 
proof of Theorem 3.3 (Case (B)) required 4.1 (*) and 4.2(1), the proof of Theorem 3.4 
(Case (C)) did not. Therefore no circular reasoning is involved because 3.4 => 4.1(*) 
for Case (C) = 4.2(1) => 3.3 => 4.1(*) for Case (B). ] 

Another example is the (real convolution operator G(x, y)=exp{ — A(x — y, 
x — y)} which, using Theorem 4.1, turns out to be bounded if and only if p < q (see 
[BL] Section 4 for more details). There is a maximizing Gaussian function if and 
only if p < q and it must have the form 


g(x) = exp{ — J(x, x) + (I, x)} (2) 
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, 


with J = (B - 7 and with / € R" arbitrary. Also 


POMET «8 aula gee ee re ae ee re oe 
= 7 (p' — q’) rar ewe ; (3) 


When p = q the limiting value C,_,, = 1/4 is correct but, since J = 0 in this case, 
there is no Gaussian maximizer. Indeed, there is no maximizer of any kind in this 
case. To prove this, note that G(x, y) = H(x — y) with H (x) = exp{ — A(x, x)} and 
J H(x — y)f(y)dy = J f(x — y)H(y)dy. Then, by Minkowski’s inequality, 


{Jf f(x — yH(y)dy|Pdx}"/? <  { [| f(x — y)|PH(y)Pdx}"/? dy 


I n/2 
= |If l,j HO)dy = () If, - (4) 


Since the condition in Lemma 3.1 for equality is clearly not satisfied, and since 
(x/A)"’? has already been shown to be the sharp bound, a maximizer cannot exist. 

A second example of a degenerate G that is bounded but does not have 
a maximizer is the following modification of the Fourier transform in R! with 
A>0. 


G,(x, y) = exp{ — Ay” — 2ixy} . (5) 


It is easily verified for all p that 2,_,(g) is unbounded on complex Gaussian 
functions when q < 2. Thus, it can be assumed that g = 2, which places us in Case 
(B) of Sect. I. If f(x) = exp{ — Jx?} is an arbitrary Gaussian function, one finds 
that when q 2 2 the optimum choice is J real and 


[2,+Ah)) = qilattip'ptipg- tia ziipe) 4. J) 3/a (6) 


By maximizing this with respect to J one finds that C,_,, is finite whenever p 2 q’ 
and C,.,,= 00 when p <q’. Ifp = q’ there is no J that maximizes the right side of 
(6) (i.e., J > 00), although the right side is bounded. Indeed, there is no maximizer of 
any kind when p = q’. If there were a maximizer f € L?(R') then, by imitating the 
proof of Theorem 4.1, it is easily seen that C,.,(G,,) > C,..,(G,) when 0 < p< 1. 
This contradicts the conclusion of Theorem 4.1 which states that the supremum 
over J of the right side of (6) correctly gives C,_,,(G,) for every A, but this 
supremum is obviously independent of A. 
These examples motivate the following theorem. 


P~P 


4.3. Theorem (a condition for Gaussian maximizers). Let G be a degenerate Gaus- 
sian kernel with the property that the n x n real, symmetric matrices A and B in (1.1) 
are both positive definite. If 1 < p< q < o then@G is bounded from L?(R") to L4(R"). 
If, additionally, p < q then Y has a maximizer which is a Gaussian function. 

If G is also real then obviously A and B must be positive definite if Y is bounded at 
all. In this real, degenerate case Y is unbounded when 1 < q < p< © and Y has no 
maximizer of any kind when 1 < p=q< ©. 


Proof. It can be assumed that G is centered and, as in the proof of Theorem 3.3, we 
can use the fact that A and B are positive definite to change variables so that G(x, y) 
is brought into the canonical form 


G(x, y) = exp{ — (x, x) — (y, y) — 2(x, Ey) — 2i(x, Hy)} (1) 
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where E and H are real matrices and E is also diagonal. In the real case H = 0. 


3 Buk a , F : 
Since M = ( E ') must be positive semidefinite, the eigenvalues e,,...,e, of 


E must be in the interval [ — 1, 1]. Since G is degenerate at least one of the e,’s (say 
e,) is +1 or —1 and, by changing y to — y if necessary, we can assume that 
e, = — 1. Thus, G(x, y) contains the factor exp{ — (x, — y,)?}. 

In the real case, H = 0, G in (1) is seen to be a tensor product of operators on R’, 
i.e., G(x, y) = G,(x,, y,)... G,(x,, y,). If p > q the operator Y, corresponding to 
e, is unbounded, as shown in 4.2, so is unbounded as well. In case p < q the 
Minkowski inequality argument in the first part of the proof of Theorem 3.2 
(applied sequentially to 9,,9,,...,9,,) shows that any maximizer, F, for 9 must 
be of the product form, ie, F(y,,...,y,) =f,(1).--A(y,) and each f must be 
a maximizer for the corresponding Y;. When p = q, however, Y, does not have 
a maximizer as stated in 4.2 and therefore Y has no maximizer. When p < q we 
know from 4.2 that each Y; has a Gaussian maximizer g;. Since [], 9;(x,) is 
a Gaussian function on R’, the proof for the real case is complete. 

For the general case with p < q, let G°(x, y) be the real kernel given by (1) but 
with H set equal to zero and let Y° denote the corresponding operator. If 
fe L?(R")L'(R") then clearly Z,_.,(G,f) < B,.,(G°, F), where F =| f|. Since 
G° is bounded when p < gq, then G is also bounded. Referring now to Theorem 4.1 
let G, be the kernel defined in that proof, ie., G,(x, y) = G(x, y)h,(x)h,(y) with 
h,(x) = exp{ — e(x, x)$, and let g, denote its unique Gaussian maximizer with 
lg. ll, = 1. Let g,(x) = uw, exp{ — (x, J,x) — i(x, K,x)} with J and K real, symmetric 
and with J, positive definite. Define g?(x) = u,exp{ — x, J,x)}. Let e +0 through 
the sequence ¢ = 1/j with j = 1, 2, 3. . . . There is a subsequence of the j’s (which we 
continue to denote by j) such that the eigenvectors of J, and K, have limits as 
j— © (because the manifold O(n) is compact). The corresponding eigenvalues of J, 
must be uniformly bounded away from 0 and oo since otherwise #,.,.(G°, g°) will 
converge to zero, as the following computation shows. 

Apart from irrelevant constants, || g° || p=! J,|~*/?, where |-| denotes the deter- 
minant. Also, || 9°(g2)||, =| J, + 1|"*|1 — E(J, + 1)” E|~*/*. Using the fact that 
|I — M™M|=|I — MM" | for any real matrix, M, we have |J — E(J,+1)~*E| 
=|I1-—(J,+1)71/7E?(J,+1)7'/?| =|J,+1|7'|J, +1 -— E?|2\|J3, + 
I\~*|J,|. Therefore 


id 


RAG, g°) <| J,pile- Hay J, ms aie ‘ni I] (Jiyle- tary Bs Ph ae ; 


i=1 


where the Ji’s are the eigenvalues of J,. Since p<q the function 
trett/P- 1/971 + t)~1/” is bounded and goes to zero as t +0 ort oo. 

The possibility that 2,_. (G°, g°) > Ois not allowed by 4.1(*) and the fact that 
R +q(G°, 9?) = By-+q(G, J.) Thus we can pass to a further subsequence such that J, 
has a positive definite limit J as ¢ > 0. This implies that y, also has a finite, nonzero 
limit p. 

The eigenvalues of K, must also stay bounded away from infinity for otherwise 
g, would tend weakly to zero in L?(R") and then the function Y(g,) would tend to 
zero pointwise. (This is so because the function y++ G(x, y)exp{ — 4(y, Jy)} is in 
L?’(R") for each x.) But Y(g,) is bounded above pointwise by Y°(g?), and the 
pointwise convergence to zero would imply by dominated convergence that 4(g,) 
converges to zero in L4(R") norm. Thus, by passing to a further subsequence, J, and 
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K, have limits J and K. From this it follows that g, converges strongly in L?(R") 
norm to g(x) = wexp{ — (x, Jx) — i(x, Kx)}. 

The Gaussian function g is the desired maximizer for Y. First note that h,g > g 
in L?(R") norm as ¢-0. Also g, > g, and thus we can write h,g, = g + A, with 
6, = || 4, ||, 0 as e+ 0. Then, since Y is bounded, 


C,+4(G) Ps R (G, g) P R (G, h,g,) ah 
Taking the limit ¢ — 0, 
Cc 


pq pq p-rqe J 


(G) 2 R,4AG, g) 2 lim sup RAG, h.g.) . 


ds. 


But by Eq. (5) of the proof of Theorem 4.1, this latter limit equals C,_,.(G). O 


4.4. Remarks and conjectures. Formula 4.1(*) gives the sharp bound. The ques- 
tion that is incompletely resolved here is whether there is a Gaussian maximizer in 
the degenerate case or, indeed, any maximizer at all. In the cases of most interest 
(e.g., Nelson’s kernel of Sect. I and the Fourier transform) the existence of a Gaus- 
sian maximizer can easily be verified by simple computation. The general case is 
algebraically complex, although Theorem 4.3 does give a criterion for a Gaussian 
maximizer and it completely settles the case of real Gaussian kernels. Indeed, as 
shown in 4.2, a maximizer need not exist even if Y is bounded. 
The examples given here lead to the following conjectures. 


pq 


(1) If there is a maximizer for cases (A), (B) or (C) of Sect. I then there is a Gaussian 
maximizer. 

(2) There is a maximizer in these cases if and only if the unique Gaussian 
maximizer g, for the mollified kernel G,(x, y) = G(x, y)h,(x)h,(y) defined in the 
proof of Theorem 4.1 has a strong limit g in L?(R") as ¢ > 0. 


Maximizers need not be unique, as shown in 4.2, but if there is any Gaussian 
maximizer for p < q then every maximizer is a Gaussian. This is Theorem 4.5, and it 
completely settles the Fourier transform case, for example. (Note that when 
p = q =2, every function in L?(R") is a maximizer for the Fourier transform and 
thus there is at least one case in which there are maximizers that are not Gaussians.) 

Theorem 4.5 also completely settles the real Case (A) because, by Theorem 4.3, 
no maximizer exists in this case when p = q and a Gaussian maximizer does exist 
when p < q. 


4.5. Theorem (when p < q, a Gaussian maximizer implies all maximizers are 
Gaussians). Let 1 < p< q< © and let G be a degenerate Gaussian kernel. Assume 
that G is a bounded operator from L?(R") to L‘4(R") and that g is a Gaussian function 
that is a maximizer for G. If f € L?(R") is another maximizer for G then f is also 
a Gaussian (but fis not necessarily proportional to g and f is not necessarily centered 
even if G is). 


Proof. Step 1. According to Lemmas 2.2 and 2.3 it can be assumed without loss of 
generality that both G and g are centered. As in the proof of Theorem 3.4, we study 
the kernel G'? = G@G. For F € L?(R7") 7 L'(R2") the inequalities (1)-(5) there 


are valid and we conclude that C,_,,(G') = (C,..,)”, where C,,, = C,..,(G). 


Step 2. If fe L?(R") is a maximizer for Y then, using O(2) invariance again, 


F(y1,Y2) = (25) 025) 


pq 
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is obviously a maximizer for Y if fis also in L'(R"), in which case F € L'(R*"). 
If f¢L'(R") consider the mollified function F,(y,,y2) = F(y;, 2) 
exp{ — (vi + Yoo Vi + y)/j} for j = 1,2,..., which is in L’ (R"). Clearly F; > F 
strongly in L?(R2") as j > oo. The function GF, can be computed as a dy, dy, 
integral of G® F; and the result (using the O(2) i invariance of G) and a change of 


variables) is 
(GF, (x, x2) = ool OE) 


with f(y) = f(y) exp{ — 2(y, y)/j}. Now the q" power integral of 9 F, can be 
computed by changing variables again and the result is || GF, = : Gf, ie Fa liq: 

However ||9 fil, IGF llq = Cp+gll fl, asi > 0 since f; > fin L'(R ") norm, and 
we conclude that |GOF i = = lim). | GF, ||, (by definition) = (C,.,)° ll F ||,» 80 


pq 
that F is indeed a maximizer for ¥°. 


Step 3. Since g is a Gaussian, it is obvious that the function z+ F(z, y) is in 
L?(R") 0 L'(R") for each y and therefore that 


K(x, y) = J G(x, z) F(z, y)dz (2) 


is well defined for each x and y in R". Since Y is a bounded operator, the function 
xt+ K(x, y) is in L4(R") for each y. We now assert that the function y+ K(x, y) is in 
L?(R") for almost every x € R” and that this function satisfies 


{JCS KG, y)itdx}4 dy}? = f { [| K(x, y)|Pdy}#/?dx , (3) 


with the understanding that both sides of (3) are finite. Formally, this assertion is 
a consequence of inequality (3) — (4) in the proof of Theorem 3.4 and the fact that 
all the inequalities (1}{5) must be equalities since F (y,, y2) is a maximizer for 9). 
If F € L'(R2") this would be correct, but if F ¢ L'(R") a proof is needed. 

Set F,(¥,,¥2) = F(y1, y2)exp{ — (V2, Y2)/f}_ for j=1,2,.... Clearly 
F,€ L' (R2*) and F;— F strongly in L’(R*") as j > 00. (Note that this F, is not the 
same one as in step 2.) Let K,(x, y) be as in (2) with F replaced by F,, so that 
K(x, y) = K(x, y)exp{ —(y, yi }. The inequalities (1}{5) in the proof of ‘Theorem 
3. 4 are then valid with F replaced by F;. As j > 00 the left side of these inequalities, 
namely || 9?) F;||4, converges to || GOP if =(C,.,)"" ll F lf =(C,.,!'Z since F is 
a maximizer. Likewise, the right side, namely (C,_.,)?*|| F, 4 also converges to 
(C,.,)'Z since F; > F. Therefore the numbers 


By = {J LJIKj(x, y)Idx}”* dy}? — ff f1Kj(x, yPdy}"dx 4) 


(which are nonnegative by Minkowski’s inequality) must converge to zero as 
j— o. Moreover, each term in B; is bounded by (C,..,)*|| F; ||} < Z, and each term 
converges to Z as j > 00 (because of inequalities (1}{5)). The first term in B; is 


a/p 
A;={JCJ|K(, yi" dx}P*exp| - 70 nay 


and, by the monotone convergence theorem, A, converges to A = (the left side of 
(3)). Therefore A = Z. The second term in B; is 


p q/p 
D,=§ ' J lK(x, nrexp} ans j nay dx . 


pq 
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The inner integral (call it E,(x)) converges (by monotone convergence) to 
E(x) = §| K(x, y)|?dy. The function E is measurable since it is the monotone limit of 
measurable functions E;. Then { {E,}*/? converges to [ {E}*/” by monotone conver- 
gence, so D, converges to the right side of (3). But, as stated above, D, also 
converges to Z, so the two sides of (3) are equal and E(x) is finite for almost every x, 
as asserted. 


Step 4. Since q > p, the strong form of Minkowski’s inequality and the equality in 
(3) implies the existence of measurable functions « and B:R" — [0, 00) such that 


| K(x, y)| = (x)B(y) (5) 


for almost every x and y in R"”. Writing G(x, y) = exp{ —(x, Ax) — 
(y, By) — 2(x, Dy)} as usual (with A and B real, symmetric, positive definite), and 
writing g(y) = exp{ —(y, Jy)} (with J symmetric and Re(J) positive definite) 
a simple computation gives 


K(x, y) = exp{ — (x, Ax) + (D’x,(B + J)~'D™x) 
— (y, Jy) }O((B + J )y — D™x) (6) 


with Q:C” +C given by 


Q(w) = exp{ —(w,(B+ J)~'w)} is(=p)ewr — (z,(B + 4J)z) + 2(z, w)}dz. 
(7) 


Evidently Q is an entire analytic function of order at most 2. Define the function 
M:R*"=>C by M(x, y) = Q((B + J )y — D*x). Plainly, since Q is entire M has an 
extension to an entire analytic function from C”" to C; call this extension N. The 
function N*:C?" > C defined by N*(x, y) = N(x, y) for x and y € C" is also entire 
analytic, and thus P = NN* is entire analytic as well. It is also true that 
P(x, y) = |M(x, y)|? when x and y are in R". From (5) and (6), 


P(x, y) = y(x)d(y) (8) 


for almost every x and y in R", and where y and 6:R” > [0, oo ) are the measurable 
functions given by y(x) = a(x)? exp{2(x, Ax) — 2Re((D’x, (B + J)~!D"™x))} and 
5(y) = B(y)* exp{2Re((y, Jy))}. If yo is a value of y such that 5(y,) + 0 and such 
that (8) holds for almost every x, we see by substituting this y, in (8) that y has an 
extension to an entire analytic function. Likewise, 6 has an extension. Thus (8) 
holds for every x and y in C" (because if two entire functions agree almost 
everywhere on R” x R" then they agree on all of C” x C"). 

Now suppose that »(x.) = 0 for some x, in C". Then, by (8), P(x9, y) = 0 for 
every yeC", which implies that for each y either (i) N(xo,y)=0 or 
(ii) N*(xo, y) = 0. This, in turn, means that for each y € C” either (i) N(x, y) = 
Q((B + J)y — D’x,) =0 or (ii) N(Xo, 9) = O((B + J )y — D™X,) = 0. Necessar- 
ily, either case (i) holds for all y in some set S < C” of positive 2n-dimensional 
Lebesgue measure ¥7" or case (ii) holds in some set S of positive #2" measure. As 
y ranges over S both (B+ J)y and (B+ J)j range over sets of positive 7" 
measure (because Re(B + J ) is positive definite and therefore Rank(B + J ) = n). 
An analytic function that vanishes on a set of positive #7" measure vanishes 
identically, and thus Q would vanish identically if y(x,) = 0. This contradicts the 
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fact that K(x, y) is not identically zero. Thus, the assumption that »(x,.) = 0 is not 
possible, and it will be assumed henceforth that y(x) + 0 for all x € C”. 

Define the set A = {y € R":d(y) + 0} c R”. This set A has positive n-dimen- 
sional Lebesgue measure ¥", for otherwise K(x, y) = 0, #2" almost everywhere. 
(In fact #"(R" ~ A) = 0 because 6 is analytic and 6 does not vanish identically, but 
this fact is not needed.) For ye A, the function Z,:C"—+C defined by 
Z,(x) = K(x, y) is entire analytic of order at most 2 and never zero (because +(x) is 
never zero). Then Z, has the form 


Z,(x) = K(x, y) + exp{ — (x, T,x) —(R,, x) + wy}, (9) 


where T, is a complex, symmetric matrix, R, € C" and yp, € C (all of which depend 
on y). I thank Eric Carlen for the simple proof of this fact, which is that Z,, being 
zero free, has an entire analytic logarithm, ic., Z, = exp{H,}. Then, since Z, has 
order at most 2, |H,(x)| is bounded above by (const.) |x|?. By a well known 
argument using Cauchy’s integral formula, H, must be a polynomial whose order is 
at most 2, i.e., Z, has the form stated in (9). 


Step 5. As noted in step 2, the function x++ K(x, y) is in L4(R") for almost every 
y € R”. By (4) > (5) of Theorem 3.4, the function z++ F (z, y) (which is in L?(R") for 
almost every y) must be a maximizer of #,., for almost every y. (Note that 
z+» F(z, y) cannot be the zero function for any y since g never vanishes.) Thus there 
is at least one point yo € R” such that 6(y,)) +0 and (9) holds and such that 
z+ F(z, yo) is a maximizer in L?(R"). Fix this y) henceforth and denote the matrix 
in (9) simply by 7. There is then a function h € L* (R") with ||h|j q’ = | such that 


J h(X)K (x, Yo)dx = I|K(-, Yo) lg = Cpgll F (+s Yop - (10) 


Since K(-, y) € L*(R"), the matrix T must satisfy Re(T) is positive definite and 
therefore K(-,y )) is a Gaussian. The optimum h satisfies h(x) = (const.) 
| K(x, Yo)|*/K(x, Yo) for x € R” and therefore h is also a Gaussian (and hence 
h € L'(R")). As remarked in step 3, F(-, yo) is in L'(R"). Therefore the function 
(x, y)++ h(x)G(x, y)F (y, yo) is in L'(R2") and Fubini’s theorem can be applied to 
(10). Thus, 


J h(x) K(x, yo)dx = f { f h(x)G(x, z)dx} F (z, yo)dz . (11) 


Since h is a Gaussian, the inner integral in (11) (call it k(z)) is also a Gaussian. Since 
F(-, yo) is a maximizer, F(z, yo) = (const.)|k(z)|"/k(z) =r(z) for almost every 
z € R". Clearly r is a Gaussian and, by (1) 


s(-738)o() = r(2) (12) 





for almost every z € R”. Setting z = w — yo, (12) yields fiw/,/2) =r(w — yo)/ 
g((w — 2y0)/./2), which is a Gaussian (in w) as asserted in the theorem. 0 


V. Gaussian kernels from L?(R") to L4(R”) 


This section consists essentially of a simple remark, but it can be a useful one in 
applications, e.g., in [L1]. Let G be a Gaussian kernel on R™ x R" with m + n, i.e., 
G(x, y) is given by (1.1) with A m x m symmetric, B n x n symmetric, D m x n and 
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Lec™*", and with M in (1.3) a positive semidefinite (m + n) x (m + n) matrix. 
Evidently Lemmas 2.1, 2.2 and 2.3 continue to hold in this case, and it can be 
assumed without loss of generality that A and B are real and L = 0. The linear 
operator Y from L?(R") to L4(R”) and the norm C,,,,(G) are defined, mutatis 


; pq 
mutandis, as in Sect. I. The remark is the following. 


5.1. Theorem (extension to m +n). Let G be a Gaussian kernel on R™ x R" as 
defined above. Then all the preceding theorems and lemmas in this paper holds, 
mutatis mutandis, in this more general case. 


Proof. Suppose m < n and extend G to a Gaussian kernel, G, on R” x R" by 
G(x, y) = h(x,)G(x>, y) 


83 x € R" is written as (x,,x,) with x,¢R" ™ and x,¢R”, and where 
h(x,) = exp{ — (x1, x,)}. Let Y be the corresponding operator from L?(R") to 
pati Note that G has the same properties as G, i.e., the degeneracy or nondegen- 
eracy of G is the same as that of G; G is in Case (A), (B) or (C) if G is; the n x n 


I 
matrix 0 4) is positive definite if and only if A is. Also, G is unbounded if G is, 


and it will be assumed henceforth that Y is bounded. | 
If fe L?(R") then evidently, as functions in L*(R"), ( f)(x) = h(x, )(Gf)(x2) 
ox thus ||9 f || pay = All zeage-m IGF lies rp"), This proves that C,_.,(G)= 
C,-+q(G) || A || peaqn-m) and that fis a maximizer for 9 G if and only if fis a maximizer for 
G. This concludes the m < n case. 
If m > n duality can be used: C,_,,(G) = Cy+p(G") where G"(x, y) = G(y, x). 
This changes the m > n case into the m < n case and, since all the theorems in this 
paper are “duality invariant”, the m > n case is proved. 


Remark. Clearly the proof of Theorem 5.1 is such that if other cases with m = n are 
settled in the future then Theorem 5.1 for m + n holds for those cases as well. 


VI. Multilinear forms in the real case and Young’s inequality 


After Sects. I to V were completed, Eric Carlen suggested that the same methods 
should yield similar results for real multilinear forms. Indeed this is so and the 
proof is outlined here (the omitted details are merely a repetition of those given 
before). Some remarks about the complex case will also be made here. Finally, 
Theorem 6.2 contains an application of the result in Sect. 6.1 for real multilinear 
forms: The truly multidimensional generalization of Young’s inequality, which was 
surmised in [BL, p. 162], will be proved. 


6.1. Multilinear forms. For i = 1,2,..., K let n; be a positive integer and let x; 
denote a point in R™. The point X = = (x,, i Xx) denotes a point in R* with 
N =F, n,. Let G(X) be a “Gaussian kernel”, i.€., 


G(x) = exp] — FF Gn Aya) +2009}. 
i=1 j=1 

where Aj; is a n; x nj matrix with A,;,= Aj, and where Le C*. The Nx N 

symmetric matrix A is the matrix whose pend are the A;,'s and G is said to be 


nondegenerate if M = Re(A) is positive definite. Otherwise M = 0 and G is degen- 
erate. 
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Let P = (p,,..., Px) satisfy 1 < p;< oo for each i. The multilinear form is 
K 
Gi... - Sg) = J G(x,,-... xg) [] Slxdx, ... dx,, (1) 
i=1 


where the integration is over R™ x R™ x ... x R"K and each fe L?(R"). The 
problem is to evaluate 


Cp = sup F(fi,.-- Sk) (2) 


where the supremum is over f;’s with || f;||,, = 1. As before, if G is degenerate we 
have to take f, € L?\(R™)  L'(R™) and then take limits. 

The cases treated in Sects. I to V correspond to K = 2 with p, = pand p, = q’. 
The case K = 1 is trivial—by Hdlder’s inequality. 

Lemma 2.1 is easily generalized to the complex, nondegenerate multilinear case; 
the details are left to the reader. The conclusion of Lemma 2.1 holds for each f in 
a maximizing set (f,, . . . , fx). The conclusions (a), (b) and (c) follow by fixing all the 
f,’s with j + i and then investigating the dependence of Y(f,,.. . , fx) on f. 

Lemma 2.2 obviously carries through as well; that is A;; can be assumed to be 
real and G can be assumed to be centered, i.e., L = 0. Likewise Lemma 2.3 carries 
through: When G is centered (i.e., L = 0) and when the supremum in (2) is restricted 
to Gaussian functions f, then each f; can be taken to be centered and, in the 
nondegenerate case, each f; must be centered. 

Let us now turn to the real case, i.e., each A;; is real and L = 0. Theorem 3.2 for 
the nondegenerate case carries through for every choice of P. The maximizing 
K-tuple (f,,...,J,) is unique (up to multiplicative constants) and each 
f(x) = exp{ — (x, J;x)} with J; being real and positive definite. To prove that f,, say, 
has this property we write (with q = p;) 


Cp =sup |4(f,--- Allg 
where the supremum is on fj,... , f¢ with \| lp, = 1 and where 


K 
SUK, . . . fee) = F CGr;, eg) PEG: ty: 
j=2 


As before, we replae 9 by 92 =GY@GJY and f,(x,),....fc(Xx) by 
F,(X2, V2), .-- > Fx(Xq, Yq) with F; € L?(R?"). To imitate the inequalities (1)—(5) in 
Theorem 3.2, define 


K 
K (X15 Y20 +++ o Vu) = J Glxy, X2,-.->Xq) [] Flxp ydx, ... dx. 
j=2 


Then, proceeding as in (1}{5) (and with the F; nonnegative for the same reason as 
before) 


IGO(F,, ..-. FS =JS OI GO, ~~ + Ye) Kei, V2 - ++ Vx) dy2- - - dy] dx, dy, 
a ) { ) G(y,,... VL) K(x1,y2; tty Yx)'dx,]'4 
x dy... dyx}idy, 
S (Cp) f {J G01... - 5 Ye) I] hj(y;)dy2 ... dyx}*dy, 
i 


(with h,(y) = [ J F(x, y'dx]"/”) 


K 
S (Cp) [] INF; lg, - 


j=2 
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As before, Minkowski’s inequality implies that 


K(X1, V2 ++ +5 Ye) = A(X, )E(V2,--- 5 Vx), 
which then implies that 


GNF, ..., Fg)(X15¥1) = A(%,)Z(y,) - 


However, 9? (F,,..., Fx) = F%~', and hence F, is a product function. The rest 
of the proof is identical to the proof of Theorem 3.2. 

By taking limits, the analogue of Theorem (4.1) hold in the degenerate case 
whenever it is known that the nondegenerate case has a Gaussian maximizer 
K-tuple. In particular, Theorem 4.1 holds in the real case. The analogue of 4.1(*) is 
that Cp is given by (2) with the supremum restricted to centered Gaussian func- 
tions. Likewise, Theorem 4.3 extends to the multilinear case under the same 
assumption about the nondegenerate case; the analogous hypothesis is that each 
A;; is positive definite. In particular Theorem 4.3 holds in the real case. 

These results can be used to derive the sharp constants in the fully multidimen- 
sional generalized Young’s inequality. Recall that Young’s original inequality 
states that if fe L?(R") and g € L*(R") then f+ g € L’(R") with 1/p + 1/g=1+4 1/r; 
here * denotes convolution. The sharp constant in this inequality was derived 
simultaneously by Beckner [B1, B2] and by Brascamp and Lieb [BL]. Another 
way to state the inequality is that 


J J AGS & — yg(y)dxdy S Clhl- NS ipaq (3) 
R" R" 
with 1/p + 1/q + 1/r = 2. The Beckner, Brascamp-Lieb result is that C can be 
determined by restricting f, g and h to be Gaussian functions. (These, in fact, are the 
only maximizers, as shown in [BL].) 

Young’s inequality (3) was generalized in several ways in [BL]. The first way is 
to allow an arbitrary number of functions f,, . . . ,f; instead of merely three as in 
(3). These are functions from R" to C and f; « L”(R"). This is Theorem 7 of [BL]. 
The integration is then over (R”)" and the arguments of the f;’s are taken to be 
((aj,x,),..., (ai, x,)) € R", where aj € R™ are specified vectors and x; € R™. Un- 
fortunately, this is not a fully mn-dimensional generalization of the n = 1 result 
because R”™" is split unnaturally into (R”)". Following Theorem 7 in [BL] we asked 
whether the full generalization is possible and Theorem 6.2 below gives it. 

A second generalization was the incorporation of a fixed Gaussian function in 
the integral, as in Theorem 6 of [BL]. Again, the Gaussian in [BL] was completely 
general when n = 1, but not otherwise. In Theorem 6.2 it is completely general. 


6.2. Theorem (fully generalized Young’s inequality). Fix K > 1, n,,...,n, and 
Pi>-++>sPx > 1 as before. Let M = 1 be an integer and let B; (fori=1,...,K) be 
a linear mapping from R™ to R". For nonnegative functions f,,...,f,, with 
f, € L?(R™) consider 


K 
I(fi,--+ Su) = 5 [LSB x)dx (1) 
pu i=l 
More generally, let g:R“ + R*, g(x) = exp{ — (x, Jx)}, be a fixed, centered, real 
Gaussian function and consider 


K 


I(fi>--- Sed = J [LKB 9(x)dx . (2) 


R” j=1 
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Let 
 alfir == ha) = 8 


and similarly for C (with I, replaced by I). Then 


C, = sup{I,(f,,---.Sk):ft. - - + »fx are real, centered Gaussian functions 
with || fil,=1}, (4) 
and similarly for C. 


Proof. Suppose the theorem is false and that the right side of (4) (call it D,) is 
strictly smaller than C,. (Alternatively, D < C.) Then there are nonnegative summ- 
able functions that are not all Gaussians, f,,...,f, of unit L” norm, such that 
I(h>- ++ >t) > D, (or I(f,, - - - sfx) > D). 

Consider the functions f{?:R" —>R* given by fi = fg! for | a positive 
integer, where g{”(x) = (I/x)"/? exp{ — I(x, x)} is an L'(R™) normalized Gaussian 
function. We note that || f{ ||, < 1 and that f{? > fin L”(R™) as | > oo. By passing 
toa subsequence (henceforth ‘still denoted by /) we can assume that f{(x) — f(x) for 
almost every x in R”. 

Evidently we can assume that M = max{n,,... , nx} and that the rank of B; is 
n; for all i. Otherwise, I or I, involves knowledge of some /; only on a hyperplane in 
R” and this means that J or J, can be made arbitrarily large (with all f,’s being 
Gaussian functions) while preserving || f;||,, = 1; the theorem would then be true in 
this case because both sides of (4) would be infinite. Similarly, the mapping 
W=J+ Fr, , BFB; (with * denoting adjoint) from R™ to R™ is positive definite; 
otherwise J, can again be made arbitrarily large with Gaussian f’s. A similar 
condition holds for I with J = 0. Since B; is linear and has full rank n;, the almost 
everywhere pointwise convergence of f{” to f, in R" implies that f{?(B,x) > f(B;x) 
for almost every x in R™. 


By Fatou’s lemma 


C, = lim inf I,(f!?,....f@) 2 1(f» --- fx) > Dy 
I+ a 


and similarly for C’ (with J in place of I,). By Fubini’s theorem, however, 


K 
(SP, - Fe) = J GPO» -- 9x) TT Mody: --- Ave - (6) 
R® = 
Here N =), n, as in Sect. 6.1, y; € R™, and G" is the centered Gaussian kernel 


GP15--- Yd = J il gi(B;x — y,)g(x)dx . (7) 
R” i=1 
Similarly, (6) and (7) hold for I in place of I, by deleting the g. (Note: Because W is 
positive definite, the integral in (7) is always finite.) 

The number C;, defined in (5) is either finite or infinite. In either case, there is 
some finite integer ‘k such that Ci =1,(f%, ....f€) > D,. However, by (6) we see 
that C; is a multilinear form as in 6.1 (1). Such ; a form has “the property, as we have 
seen in Section 6.1, that its supremum over f’s with || f;||,, = 1 is equal to its 
supremum over real, centered Gaussian functions. But if we set all the f’s equal to 
Gaussian functions we have that f(’s are also Gaussian functions and || ff” i,m 4% 
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This means that Cj < D,, and this is a contradiction. The same proof holds for J in 
place of J,. O 
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The aim of this paper is to study a certain type of chamber systems and give 
application to the classification of geometries whose diagram is a Coxeter dia- 
gram. In this sense this paper is a successor of [13]. For notations and definitions 
concerning chamber systems see [13] or [19]. For convenience of the reader 
we recall two definitions which are of importance for this paper. Let @ 
=(@, (¥;);-;) be a chamber system over I and ce@. With A,(c), J SI, we denote 
the element in $3, containing c(4,(c) is a chambersystem over J). Let A be 
the diagram of @, JCI, then A, is the diagram which consists of all nodes 
of A not in J and all those bonds which do not contain some element in J. 
We are going to prove 


Theorem. Let @ be a classical, locally finite, nontight Tits chamber system of 
characteristic 2 over a set I with connected diagram A=A(I) of rank=3, and 
with transitive automorphism group G and finite chamber stabilizer B=G,(ce@). 
For JSI let G, be the stabilizer of A,~,(c) and K, be the kernel of the action 
of G; on A,_,j(c), X;=G4,4c), i€ I. Suppose furthermore that A contains no bonds 
of strength >2 but at least one bond is of strength 2. Finally if JSI such that 
A,_,(c) belongs to o——=D——O or O—-O— we assume that G,/K,# A>. 
Then one of the following holds. 

(1) @ is the chamber system of a finite spherical building and G is an extension 
of a group of Lie-type A by diagonal and field automorphisms. 

(2) A is a complete bipartite graph containing no single bonds and one of 
the following holds 

(a) If A, ,(c) belongs too——othen G,/K,= Ag or Sg. 

(b) If A,_,(c) belongs to o——o then G,/K,=0,(2) or O,(2). Furthermore 
A is a star. 

(c) There is a distinguished point 1 in A such that if 1eI—J and A,_,(c) 
belongs to O——o , then G,/K,=~0,(2). If 1€1—J and A,_,(c) belongs to 
O—», then G,/K,= Ag or Sg. 

In any case there is some J such that A,_,(c) belongs to c——o-——D 
(leI—J in case (c)) such that K,=1. 
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(3) |I|=n>3 and A is of type 
1 


m2=2, m+1<n 


™ 


For J=I—K, K C(I, ..., m}, |K|=m-—1, we have K,=1, 4,_,(c) is a building 
and G, is an automorphism group of the corresponding group of Lie type. 


(Here and later on we denote by U,(2) any subgroup of Aut(U,(2)) containing 
U, (2)). 

It has been shown in [20] that a chamber system with nonconnected diagram 
is the “direct sum” of chamber systems with connected diagrams. So this assump- 
tion in the theorem is not a real restriction. 

As any parabolic system in the sense of [19] yields a chamber system with 
the same diagram, we get a similar theorem for parabolic systems, which can 
be worked out as an exercise. 

Suppose now I is a geometry over J. Then one can define the chamber 
system @(I) in the following way: The chambers are the flags of I of type 
I, two such chambers being i-adjacent if and only if they coincide in a flag 
of type I— {i}. Then @(I) is connected if I is residually connected. Furthermore 
from [19] we get that if I is a classical Tits geometry over J with diagram 
A and flag-transitive automorphism group G, then @(I) is a classical Tits 
chamber system over J with the same diagram A and G is a chamber transitive 
automorphism group of @(I). 

If (I) has characteristic p we say that I’ is of characteristic p. 


Now we get with our theorem and the results of [3, 12] and [14] the following 
corollary: 


Corollary. Let I be a finite classical Tits geometry of characteristic 2 over I, 
|I|2=3, with connected diagram A = A(I). Suppose that A contains bonds of strength 
2 but no bonds of higher strength. Suppose that T admits a flag transitive 
automorphism group G. For JSI denote by G, the stabilizer of the residue I; 
of a flag F of type J and K, the kernel of the action of G, on I. Then one 
of the following holds. 

(1) I is a finite building and G is an extension of the corresponding simple 
group of Lie-type by diagonal and field automorphisms or A,, or 
A= O—O— and G2 A). 

(2) A is a complete bipartite graph, all of whose bonds are of strength 2 
and @ (I) is as in Theorem (2). 

(3) A is of type 

1 
m=2, n+m+1 


™ 


and one of the following holds 

(a) @(L) is as in Theorem (3), or 

(b) n=m+2, Gyo, m is an extension of an elementary abelian group of order 
64 by A; and G,=Sp,(2) for any JC {I1, ..., m}, with |J|=m—1 and leJ. 





Chamber Systems 
(4) A is one of the following 
, } { | | ) { | | p3 i | y) 
If G; belongs to o——0——-O or O_O then G, A3. In the first 


two cases G= U;(5). 
(5) A is of type 























G,=G, is an extension of an elementary abelian group of order 16 by A, and 
(6) A is of type 


G,=G,=Sp,(2) and G, is an extension of an elementary abelian group of order 
64 by A. 
(7) A is of type 


For any J&{4, ..., n}, |J|=n—4, we have that G, is as in (6). In particular 
Kyo a)=Kyc)=1- 


Similar results for classical, locally finite Tits chamber systems of characteris- 
tic p> 3, may be found in [19]. Thus the only remaining case is just p=3. 

In fact all the possibilities in the theorems and the corollary really occur. 
We restrict ourselves to the corollary. Examples of type (2) and (3) have been 
constructed by Aschbacher, Kantor, Meixner, Smith or Wester. Some of them 
may be found in Habilitationsschrift of M. Wester [21], in [8], or [1, 4]. For 
n=3 we have interesting examples like U,(p) [5] and Suz [9]. The last two 
diagrams in (4) can be realized in Q; (3) see [21]. Type (5) can be realized 
in McLaughlin’s simple group [11]. Type (6) can be realized in Q,(p) [1]. 
Type (7) can be constructed using the ideas of [8]. 

Now some words about the proof of the theorem. By [13] we may assume 
that 4 contains co——co——r . Now we argue by induction. If rank(4)=3, 
then the result follows from [18] (see (1.8)). So we first assume that rank(4)=4. 
This gives us a very restricted list of possible diagrams. Let ie/J, such that 
A; is connected then we get that 4, _,,(c) is either a building or a chambersystem 
belonging to 0——0——> and by assumption the latter occurs. Now in most 
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cases the structure of the groups belonging to the latter chambersystem as 
described in (1.8) yields that the groups of Lie type (if there are any) are defined 
over GF(2). This then gives that the kernels K, are 2-closed, mostly even 2- 
groups. Now G,/K; acts on K; and there are two cases to consider: C,(K;) 9K; 
and €,(K,)& K;. 

If one checks the groups described in the theorem for rank (A) =4, one realizes 
that the generic case will be K;=1. This means that €,(K,;)9/K; will lead to 
the diagrams and groups as in the theorem. Now in this situation it turns 
out that just by comparing the structures of different G,s, it is an easy job 
to get K;=1. 

So assume that €,(K,)<K; for the “relevant” G,’s. Let SeSyl,(B), then 
set Z;=<Q,(Z(S))%> <K;. Now G; acts on Z;. We are able to determine the 
action of G; on Z; for different j’s. It then turns out that these actions do 
not fit together, giving a contradiction. As an example for this see (1.12)+(1.14). 
It is worthwhile mentioning that the G,’s have not to be finite. Nevertheless 
we are able to prove properties of finite representations for such groups. 

After having treated the case rank(4)=4 the remainder of the proof is just 
an easy exercise in induction. We just have to pin down all diagrams where 
any connected subdiagram containing a double bond is one of the diagrams 
described in the theorem. Just by combinatorics and the fact that rank(4)>4 
we get that A is one of the diagrams described in the theorem. 

For the proof of the theorem in Chap. 2 and 3 we fix the notation of the 
theorem. If J={a, b,c ...} we also write G,,,... and K,,,... instead of G, or 
K,. Furthermore we set Z, = <Q, (Z(S))®>, where SeSyl,(B). 

If G is a finite group V a faithful GF(2) G-module, A+1 an elementary 


abelian 2-subgroup of G with |V: €,(A)|<2'|A| for some ieNv {0}, then V 
is called an F;-module with offending subgroup A. An Fo-module is also called 
an F-module. All other notations will be standard. 


§ 1. Preliminaries 


For convenience of the reader we summarize some of the preliminary lemmas 
contained in [13]. 


(1.1) Lemma. Let G be one of Sp2,(q), Q>(2n, q), (S)U,(q), n=3, q=2/, and 
V an irreducible GF (2) G-module. If V is an F,-module, then 

(a) G=Sp(2n, q), V is an F-module and V is the natural module or n=3 
and V is the spinmodule, or V is not an F-module n=4, q=2 and V is the 16- 
dimensional spinmodule 

(b) G=Q™-(2n, q), V is the natural module, 

(c) G=(S) U,(q), V is the natural module, or n=4 and V is the natural module 
for Q¢ (q) (= U,(q)). 


Proof. [2; (2.1)], [6; (6.40)]. 


(1.2) Lemma. Let G=Sp(2n, q), q=2/, V an irreducible GF(q)-module, G 
=<¢X,,...,X,)>, where the X;’s are the minimal parabolics containing a fixed 
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Borel subgroup corresponding to the following diagram 


Let SeSyl,(X ,). 
(a) If V is the natural module, then 


Ng(Cy(S))=< Xi, tees Xn-1> 


(b) If V is the spinmodule, then Ng(Cy(S))=< Xo, ..-, Xn>- 
If q=2 the converse of (a) and (b) is true. 


Proof. Well known [10]. 


(1.3) Lemma. Let GYQ™(2n, q), (S) U(2(n—1), q) or (S) U(2n—1, q), q=2/, G 
=<X,,..-, Xn—-1)>, where the X;’s are the minimal parabolics containing a fixed 
Borel subgroup corresponding to the following diagram 


1 2 n-1 


For n=3 choose notation such that O? (X ,/O,(X ,))=L2(q?) for Q™ (6, q) and 
O?'(X,/0,(X 2))=L>(q’) otherwise. Let V be an irreducible GF (2)-module for G 
and SeSyl,(X,). Then 

(a) If V is the natural module, then Ng(Cy(S))=<X 1, .--, Xn-2>- 

(b) If G=U,(q) and V is the natural Q¢ (q)-module, then Ng(Cy(S))=X >. 


(1.4) Lemma. Let G be Q-(2n, q), q=2/, n>3. Denote by G the stabilizer of 


a singular point in the natural representation of G. Let V be a nontrivial GF (2) 
G-module and 1+ASO,(G). If V is an F-module, then A is not an offending 
subgroup. If V is an F,-module and A is an offending subgroup, then q=2. 


Proof. An easy calculation shows |V: Cy(A)|=q|A| for any 1+ ASO,(G). 


(1.5) Lemma. Let G=<G,, G,>, K;G,, | K;|< 0, i=1, 2, and K, ©G,, K, SG. 
Suppose there is no nontrivial normal subgroup N of G contained in G, OG). 
Let p be a prime with p¥|K, K,/K,|, p¥|K,K,/K>|, then p¥|K, K3|. 


Proof. [13; (1.12)]. 


(1.6) Lemma. Let @ be a connected chamber system over I={1,...,n}, n22, 
with chamber transitive automorphism group G. Fix a chamber c and set B=G,. 
Assuming that B is finite. Set X;=G,,(.. and let Q; be the kernel of the action 
of X; on A,(c). Suppose that B/Q; is a 2-group for i=1,...,n, then also B is 
a 2-group. 


Proof. (13; (1.13)]. 


(1.7) Lemma. Let G=<G,, G,> with |G,|< 0 and 
(i) F*(G,)=0,(G,). 
(ii) G; AG, =B=N,,(S), SeSyl,(G, 0 G2). 
(iii) No nonidentity subgroup of S is normal in both G, and G,. 
(iv) 07 (G,/02(G2))=S3. 
(v) Z,SC(Z,)=aG,, where Z;=<Q,(Z(S))®, i=1, 2. 
Then either 
(1) Z, is an F,-module for G,/Cg,(Z,) with offending subgroup in S/Cs(Z,), 
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or 
_ Z,+2Q,(Z(S)), Z;Z2<1G, and if peO* (G,), o(p)=3, then |[p, 02(G2)]| 


Proof (cf. [16; (2.1), (2.2)]). (Unfortunately, [loc.cit.] also assumes F*(G,) 
=0,(G,) but what is really used is that C5(Z,)<1G,, which we assume to be 
true by (v).) 


(1.8) Lemma. Let @ be a classical locally finite nontight Tits chamber system 
of characteristic two with diagram A of type } : 3, and transitive 
automorphism group G with finite chamber stabilizer B=Gc. For ieI = {1, 2, 3} 
let G; be the stabilizer of A,~,,(c) and K; the kernel of the action of G; on 
A,-(c) and Q;=0,(K;). Then one of the following holds: 
(i) G,/K;2S¢ or Ag, i=1, 3. Cg,(Kj) SK; for i=1, 3, 2* <|K;|S2°, Z(G) =1, 
i=1, 3 and Q,(Z(S))\aG,, SeSyl,(B). 
(ii) G;=G3=S, or Ag and K,=K3=1, G,/K,=(Z; x Z;)-2. 
(iti) G, = G3=Q¢ (2) or O; (2), Ky =K3=1, G, NG; involves As. 
(iv) G,/K,=Q¢ (2) (or O¢ (2)), G3/K3= Ag (or S,), K, is extraspecial of order 
2’, K, is 2-closed with |K,:Q3|=3, G3/Q;%3A, (or 3S.) and Q, is special 
with |Z(Q3)|=2+, |Q3/@(Q3)|=2°. Furthermore Q3/(Q3) and $(Q;) are irreduc- 
ible G3/Q-modules. K, is special of order 2'° with |Z(K)|=4 and G,/K,=A; 
x S; (or S; x S3). 
(v) GP [GP =U, (2), K; =K3=1, G, NG; involves As. 


Proof (18; Theorem 1]. 
The next lemma is in the spirit of [7]. 


(1.9) Lemma. Let G be as in (1.8)(i) and V be a finite nontrivial irreducible GF (2) 
G-module, then there is no WE K;,, i=1 or 3, |W|=4 such that [V, W, W]=1. 


Proof (see [7; (3.2)]). Denote by Q;, i=1, 3, the subgroup of K;, such that |Q;|= 16 
and 0,<G;. By Q, we denote the extraspecial normal subgroup of G, of order 
32. We also denote Q, by Q, where <z) =2,(Z(S)). 

We will show that we may assume WCQ,. So suppose not. Then | K;: Q;|=2 
and |WnQ;|=2. We may assume that WOAQ;=Q,(Z(S)). Furthermore K; is 
an indecomposable module for G,/K; and so G,/K; possesses orbits of length 
6 and 10 on K;\Q;. Let weW\Q,, then in both cases Cg, (w) acts irreducibly 
on Q;. Hence [[V, w], Q;]=1. In particular any foursgroup in K; containing 
w acts quadratically. Let teS\Cs(w). Then [w, t]=xeQ; and Cog,(t)+<x>. 
Choose yeCo,(t)\<x>. Then <y, w> acts quadratically and <y, x w> acts quadrat- 
ically. Hence also <y, x> <Q; acts quadratically. So we assume: 

(1) W<Q;. 

Now the action of G; on Q; shows that we may assume 

(2) W=0,093. 

As C,,([z, V])a1G2, we get [Q>, [z, V]]=1. 

(3) LV, Q:, Q;, O?(G)] =1, i=1, 3. 

We have |Q;: 0;AQ,|=2. Choose geG; such that Q;=<z®>(Q;0Q,). Then 
<Q., 0.6 0.28) P 07(G)) and hence LV, Zz, Q;] ee LV, Zz, 2°] <C,(07(G;)). 

(4) [C,(Q)), 92, Q2.]=1, i=1, 3. 

Let heG,\G;, then Q,=(Q.9Q)(Q2.7Q}) and thus [C,(Q), Q2, Q2] 
<$€,(Q) LV, Qi, QS C,(<Q;, 07(G)">)_ by (3). Since C,(G)=1 and 
0? (G2) S<Q;, 07(G;)">, we get G=S<Q;, O?(G,)"> and so [C,(Q)), Q2, Q2]=1. 
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(5) [C,(Q)), Q;, Qj] = 1 where ti, i} anes {1, 3}. 

This follows from Q;=(Q;0Q2)(Q;0Q;)* for suitable geG; and 
LV, Q:0Q;, Q:0Q,]=1 by (2). 

Now choose notation so that [C,(S), O?(G,)]+1. Then on C,(Q,) both 
classes of fours-groups in 07(G,)/Q,~A, operate quadratically by (4) and (5). 
But A, does not possess such a module. This contradiction proves the lemma. 


(1.10) Lemma. Let G be as in (1.8)(i) and V be a finite irreducible nontrivial 
GF(2) G-module. Set L;=O7(G,) and Q;=O,(L,), i=1, 3. Furthermore set Q, 
=K,L,. Choose notation such that [(C,(Q,), L,]+1. Then the following holds: 

(1) |V: C,(Q,)|=2° and L, acts nontrivially on V/Cy(Q;). 

(2) If teQ;*, i=1, 3 then |[V, t]|=16. 

(3) If K;>Q,, for i=1 or 3, then |[V, t]|=64 for teQ}. Furthermore |[V, t]| 
>2° for teK;\Q;. 

(4) If seG,\K,, then |[V, s]|28. 

(5) There is no elementary abelian subgroup A+1 of S such that |V: C,(A)| 
<2|A| (i.e. G possesses no F,-module with offending subgroup contained in S.) 


Proof. As [V,Q,,Q,]+#1 by (1.9) we have [Q,,V/C,(Q,)]+1. This shows 
|V: Cy(Q,)|=2° which is (1). 

Furthermore A, ~L,/Q, induces a nontrivial composition factor in V/Cy(Q;) 
and €,(Q,) and so |V: C,(t)|216 for any teL,\Q,. But all involutions in 
Q, UQ; are conjugate to some involution in L,\\Q, which gives (2). 

Suppose next K;>Q; for some i=1 or 3. Let first teK;\Q;, then 
C,(O=E,¢As Or Ey6(Z;xZ3)Z,, where O,(C,,(t)) is an _ irreducible 
C,,(t)-module. If |[V,t]|<16, then the structure of GL(4,2) shows that 
[0,(C,,(), LV, t]]=1. This implies that any WC K,, |W|=4, containing t, acts 
quadratically on V. This contradicts (1.9). So we have |V: €,(t)|=2°. 

Let now teQ,. We may assume that teZ(G,). As K;>Q;, there is WCK,, 
|W|=4, teW and WaG,. Now W acts on [V,t]. As there is no involution 
in GL(5, 2) centralized by an elementary abelian group of order 9, we get 
LLY, t], W] =1 if | LV, t]| $32, contradicting (1.9). This proves (3). 

For proving (4) choose seG,\L, with |[V, s]|<4. There is some teQ} with 
s~st. Hence |[V,t]|<16 and so by (3) Q,=K,. Now either |Cg,(s)|=8 or 
|Co, (s)|=4 and there is some peG,, o(p)=3, [p, s]=1 and (C9, (s), p] =Co, (5). 
As |[V, s]|<4 we get in both cases a subgroup W of order 4 in Cg,(s) such 
that [W, [V, s]]=1. Hence <w, s> acts quadratically on V for any weW*. Now 
choose geQ, with s*=st+s. Then there is some weW*, w+t. Hence <w, st) 
and <w, s> act quadratically, but then also <w, t> acts quadratically, contradict- 
ing (1.9). 

Assume (5) is false. Choose ACS, with |V: C,(A)|| $2|A|. By (2) and (4) 
| A|=8. We provide a contradiction in a series of steps. 

Suppose AN Q,=1. Then |A|=8 and An K, =1 by (3). Set W=ANL,. Then 
|W|=4 and by (2) €,(A)=C,(a) for any aeW*. Hence W acts quadratically 
on V. There is some geL, such that teW*, teZ(S). Hence WC L,0G,. Now 
W§cC,,([V. t]aG,. This shows Q,¢C,,([V, t]). But then for seQ,\<t, 
s€Q,, we have that <s, t> acts quadratically on V, contradicting (1.9). So we 
have shown 

(a) ANQ, +1. 

Suppose AGQ,. If AC K,, then by (3) A=K,, and C,(t)=C,(A) for any 
teQ;. But then Q, acts quadratically on V, contradicting (1.9). So we have 
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AQGK,. By the choice of L,, we have [A, Cy(ANQ,)]+1. Furthermore by (2) 
|V: Cy(AnQ,)|=16. This shows |A|216. If K;+Q,; for some i, then by (3) we 
even get |V: C,(ANQ,)|=64 and so |A|= 64. But there are no elementary abe- 
lian subgroups of order 64 in S. So we have 

(b) Either ASQ, or Q;=K;, and |A|216. 

Suppose again AGQ,. If |V: Cy(ANQ,)|= 32, then we get |A|=32 and so 
|AQ,|=4. Now by the choice of L, we have that |C,(Q,): Cy(Q,) 1 C,(A)| 
>4. Then |C,(ANQ,): Cy(A)|=4 and so |V: C,(A)|=2’, a contradiction. Thus 
we have |V: C)(ANQ,)|=16. Now |ANQ,|=2 and so again |A: ANQ,|=8. 
Hence again |C,(A NQ,): C,(A)|=4 which gives |V: C,(A)|=64=4|A|, a con- 
tradiction. So we have 

(c) ACQ,. 

By (2) |A|28. If |A|=8, then A acts quadratically by (2), contradicting (1.9). 
So we are left with A=Q,. Now |V:C,(Q,)|=2°. Furthermore by (1.9) 
|V: C,(t)|=2* for any teQ;*. Hence again by (1.9) we have 

(d) C,(Q,)=C,(W) for any foursgroup WC Q,. 

Choose geG, with |0§G,|=8. Then |Q%: 0, .0%|=4. By (d) €,(Q;) 
=C,(0§ 00,)=C,(Q0§). Now C,(Q,) is invariant under <G,, G%>. As 
G,5<G,, G§>, we have <G,, G{> =G, a contradiction. 

For groups of type (1.8)(iv) there is no result like (1.9) as 3-Suz is a group 
of type (1.8)(iv) possessing quadratic foursgroup. Anyway we prove 


(1.11) Lemma. Let G be as in (1.8)(iv) and V be a finite nontrivial irreducible 
GF (2) G-module. Set L;=O7(G,), i=1, 2, 3. Then the following holds 

(1) LV, Z(G,), L,J +1. 

(2) |LV, Z(G,)]|22°. 


(3) There is no elementary abelian subgroup A in S such that |V: C,(A)| 
S4/Al. 


Proof. Let [LV, Z(G,)], L;]=1. Then any foursgroup in K , which contains Z(G,) 
acts quadratically on V. Now choose géG,\G, such that Z(G,*CK,. If 
C,(Z(G,))=C,(Z(G,)), then C,(Z(G,)) is invariant under <L,,g>S 
=<L,, L,> S=G, a contradiction. Hence C,(Z(G,))+C,(Z(G%)) and so as all 
nontrivial subgroups in <Z(G,),Z(G,*®> are conjugate under 
N({<Z(G,), Z(G,)®>) we have W=[V, Z(G,)] OV, Z(G,)?] +1. Now W is invar- 
iant under <L,, [?,> >L,, a contradiction as before. This proves (1). 

As L, involves Q¢ (2) (2) is a consequence of (1). 

Now assume that there exists an ACS with |V: C,(A)|<4|A|. We provide 
a contradiction in a series of steps 


(*) ANK,=1. 


Suppose false. As all involutions in K, are conjugate to the involution in 
Z(G,), we get |V: €,(A)|=2° with (2). In particular |A|=16. Without loss 
Z(G,)SA. As m,(K,)=3, we have ASK, and so by (1) and (1.1) 
ILV, Z(Gy)]: Cy, 26 (A)|2Z1A: ANQi I. In particular IC, (Z(G,)): 
Cey-iz1G,(A)|Z/A: AQ, |. By (2) |V: Cy(Z(G,)|Z2°. As Z(G)SA we get 
4|A|=|V: C,(A)|22°-|A: ANQ,|. This now gives |ANQ,|=16, a contradic- 
tion. 
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By (1) and (*) A acts nontrivially on [V, t] and V/C,(t), <t)=Z(G,). Hence 
|V: Cy(A)|=|A|? and so |A|<4. Now A contains a transvection a on [V,t] 
since A is offending on [V, t]. As C,(t)=[V, t] C,(A) we obtain |V: C,(a)|=4. 
(Note that Cg, /x,(aK,) acts irreducibly on C;(a)/[V, a], V=V/C,(t).) As aeG;, 
there is some xe€Z(K3) such that a~ax in G,. Hence |V: C,(x)|<16. As <x) 
~ Z(G,) in G, we have a contradiction to (2). 

Next we prove a lemma which is based on the interaction of (1.7), (1.10) 
and (1.11) and which will be one of the main tools for the proof of the theorem. 


(1.12) Lemma. Let @ be a classical locally finite nontight Tits chamber system 
of characteristic two of rank four with transitive automorphism group G and 
finite chamber stabilizer B=G,. For i¢I={1, 2, 3,4} let G; be the stabilizer of 
A,~,)(c), K; be the kernel of G; on A;_,,(c), and X; the stabilizer of A,(c). Denote 
by S a Sylow 2-subgroup of B. Suppose furthermore 
(i) G,/K, is as in (1.8)(i) or (1. 8)(iv) with K, a 2-group and (;(K,)&K,. 

a There is X,<X,<G, with X,/0,(X, yx S;, SeSyl,(X,), where G,/K, 
is as in (1.8)(i), (1.8)(iv) or G,/K,=Sp,(2). 

(iii) <G,, X,>=<G,, X,>=G (in fact (iii) follows from the other assumptions). 

Then 2, (Z(S))aG,. 


Proof. Suppose false. We are going to show that we have the assumptions of 
(1.7) for the pair (G,, X,). 

By assumption B is finite and so S is finite and then by (ii) X, is finite. 

As X, K,/K, is contained in a parabolic of G,/K, we get from (1.8) resp. 
the structure of Sp,(2) that F*(X ,/K,)=0O,(X ,/K>). So we have (1.7)(i). 

By (ii) and (iii) G, AX, =S= Nx, (S) and so we have (1.7)(ii). As ¢G,, X»)=G 


we also have (1.7) (iii). Further (1.7)(iv) follows from the definition of X,. 

Set Z, =<Q,(Z(S))%'). Then by assumption Z, +2,(Z(S)). As C5(K,)SK,, 
we have Z,°Z(K,). Since by the structure of G,/K, described in (1.8) there 
is no normal subgroup N of G,/K, with NO S/K, +1 we have €,(Z,)=K,1G, 
giving (1.7)(v). 

Now we may apply (1.7) and so we get two possibilities. If Z, is an F,-module 
with offending subgroup in S/K, we get a contradiction to (1.10) or (1.11). 

So we may assume that for pe X,, 0(p)=3, we get |[0,(X,), p]|=4. But 
now an easy inspection of the groups in (1.8)(i), (1.8)(iv), Sp,(2), respectively, 
shows that |[0,(X,K,/K>), p]|=16. This proves the lemma. 


(1.13) Lemma. Let @ be a classical locally finite Tits chamber system of rank 
four with transitive automorphism group G and finite chamber stabilizer B=G.,. 
For ie{1, 2,3,4}=I let G; be the stabilizer of A,~,,(c) and K; be the kernel 
of G; on A,_,,(c). Assume further that K, and K, are 2-groups and the following 
hold: 

(i) G,/K,=Sp,(2) and G,/K, is as in (1.8)(i). 

(ii) G,; \G,/K, is the stabilizer of a point in the natural representation of 
Sp, (2) (i.e. Gy A G2/K,=2\2*\S,). 

(iii) Q,(Z(S))aG,, where SeSyl,(B), Cs(K,)SK;,. 

Then there exists a Z,<1G,, Z,;,SK, such that Z, is the natural module 
for Sp¢(2)=G,/K,. 
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Proof. The structure of G, in (1.8)(i) shows G, = <S%) and so 2,(Z(S))a1Z(G,). 
Set Z, =<Q,(Z(S))™>. Then Z, 1 Z(G,)=1. Let X, be the parabolic in G, con- 
taining S but not contained in G,; NG). 

Now we are going to apply (1.7) to the pair (G,, X,). We have |X,|<o 
and as X,/K,<G,/K, we see from (1.8)(i) that F*(X,/K,)=O,(X,/K,) and 
so F*(X,)=0O,(X,). As X,/0,(X,)=S3, we have X, 1G, =SeSyl,(G; 0X). 
So we have that (1.7)(i)}{iv) are satisfied. 

As (¢,(Z,)aG,, we get €,¢,(Z,)=G, or K,. But Z(G,)=1, and so 
Cg, (Z,)=K,. 

Now (1.7) implies that either Z, is an F,-module or there is some peX,, 
o(p)=3, such that |[0,(X,), p]|=4. But as X,<G,, we see from (1.8)(i) that 
|[0,(X,/K,), p]|216 for any p, and so we have that Z, is an F,-module. 

By (1.1) Z, contains a submodule Z, which is either the natural module 
or the spinmodule. As G, 0G, centralizes 2,(Z(S))\Z,, we get with (1.2) that 
Z, is the natural module, which is the assertion. 

For the proof of the next lemma we need the concept of the so-called coset 
graph. We are going to give definitions and some properties. For more informa- 
tion see [16]. 

Let H be a group and H,, H, be two different subgroups of H with H 
=<H,,H,>. The graph [=I°(H,, H,) has as its vertices all the right cosets 
of H, and H, in H, and two cosets are adjacent if they are distinct and have 
nonempty intersection. The group H acts on I by right multiplication. Two 
vertices 6 and J of I are said to be conjugate (6~A) under H if they are in 
the same H-orbit of I: With d(6, a) we denote the usual distance metric on 
I and A(5)={aeT: d(d, «)=1}. The following elementary properties can be easi- 
ly checked: 

(i) I is bipartite and connected. 

(ii) H&Aut(L) iff there is no non-trivial normal subgroup of H contained 
in H,OH3. 

(iii) H operates edges-transitively on f° 

(iv) The vertex-stabilizers H;, d€Tj are conjugate in H to H, or H3. 

(v) H; is transitive on A(6) for dE. 

(vi) The edge-stabilizers are conjugate in H to H, V0 H. 

Usually we identify the vertex H, with 1 and H, with 2. If X; is subgroup 
of H;, i=1,2, «eI, a~i, then we denote by X, the corresponding subgroup 
of H,. 


(1.14) Lemma. Let the assumptions be as in (1.13). Then C,(K,)9K3. 


Proof. We choose notation as in (1.13). Then there is a subgroup Z,<1G,, such 
that Z, is the natural module. Assume €,(K,)GK>. We have G=<G,, G,). 
Let X, be the parabolic in G, with X,9G,9G,. Then G=¢X,, G,>. To sim- 
plify notation set H,; = X,, H,=G,, 0; =0,(X>2), 0, =K>. To prove the lemma, 
we use the coset graph I'(H,, H,). 

Let zeQ,(Z(S))* 1Z,. Then as Q,(Z(S))=aH>, we get zeZ(H,). Set T, 
= z*2), then |T,|=4 and T,<1H,. Set T, =<z). 

Let b=min {d(1, «)|ael, T, SH,, T, JH, for some Be A(a)}. 

For the remainder of the proof let (1, «) be a pair, with d(1, «)=b, T, <H,, 
T, SH, for some Be A(x). 
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(1) Suppose US Z(H) Q>2, or U=(Q,). Then Cy, ,4,(Q2/U)SQ>. 

For any parabolic P of H,/Q, we have that Cy,(Q./U) Q,/0,.1PaP. So 
we see with (1.8)(i) that O?(P)<Cy,(Q2/U) Q2/Q,0 P for any P or the intersec- 
tion is always trivial. In the latter we have the assertion. So assume that O7(P) 
centralizes Q,/U, then O?(P) centralizes Q, and so €;(Q,)9Q),, a contradiction. 

Set V,=<Tf?>. | 

(2) If V,.5Q, then [Q2, V,]ST, and Cy, ,4,(V2/T,)=Q2. 

We have T=H, 7H, €Syl,(H,) and [T; T,]=T,. Thus [Q,, T,] ST, and so 
[Q,.,V.]ST,. As for any parabolic subgroup P of H,/Q, we have that 
Cy, (V2/T,)/Q20 P<: P, we see with (1.8)(i) that Cy,(V2/T,)/Q, 1 P=1 and so 
Cy, .#2(V2/T,)=Q2. 

(3) b22. 

By (2) |[T,, Q2]|S2. Application of (1.10) (5) gives [T,, 0,/6(Q2)]=1 and 
so by (1) T, © Qo, i.e. b2=2. 

(4) b=2 in particular V,<Q,, V2.9 Q. 

Assume b= 3. Then V, is abelian and V,<Q,. 

We claim a~2. So suppose «~1. Let (1, 2, 3, ....«—3,«—2,«—1, a) be a 
path from 1 to « of length b. Let Oe A(i)—{2}, then V)CH,_,. As T,_2 
=T,_3;T,-, and T,_,S7,, since a~1 and so T,°H,, we get [M, T,-;] 
=[V, T,-2]=1. This shows yp) ©H,_, 0 H,-2. Now 


[V,~10 Ho, Vo, Va-1 0H SLVa-1 OH, Vz- 1 AH] =1. 


Hence V,_,;QH )OH;,/Koy acts quadratically on he and so by (1.9) 
[V.-1 0H: Va-1 VQo|S2. As H,/Q; =S3, we get |V,-1:V,-1 VQo|S4. 

If [(V.-,; Qo, Vo] #1, then T>=[V,-,9Qo, — VoJSV,-, and so 
[V1 To T2]=1, i.e. [T,, T,]=1, a contradiction. Thus |V,_,: Cy__,(t)| $4 for 
any teV,*. By (1.10). .Yy<Q,-,SH,. As H,/Cy(T,)=S; we get [T,, Vo] 
=[T,, T,]=T,& Vo. Hence Vy <T,, Hp> = <H,, Ho> =G, a contradiction. This 
proves the claim. 

Now a~2 and so V,CH,, V,SH>. Thus [V2, V,J SV, 0 V,. Again (1.9) gives 
|V,: V,z1Q2|S2 and so |V,: Cy_(T,)| <4. As T, JQ,, this contradicts (1.10) and 
hike. (V,/T,) =Q, by (2). So (4) holds. 

Up to now we never used the action of G, on Z,. Suppose Z,<Q,. By 
(4) V;<Q, and by (2) [V,, Z,] ST. In particular V, K ,/K, induces transvections 
on Z,. As V,K,/K,;&G,G,/K,, we have that V, K,/K, © Z(G, \G,/K,). But 
then, as H,/K, is a parabolic of Sp,(2)=G,/K, containing a Sylow 2-subgroup 
of G, 1G,/K,, we have V,<Q,, a contradiction to (4). 

So Z,9Q). By (1.8)(i) we have |Z,: Z,; 1Q,|=16 or 32 and so |Z, 0 Q,|=4 
or 2. But the action of Sp,(2) on the ontnuel module implies that G,; 1G2/K, 
does not normalize a group of order 4 in Z, and so |Z, 9Q2|=2, |Z,: Z; 1Q2| 
=32. Hence Z,\1Q,=T, and [Q,, Z,]<0,0Z,=T). As T,2Z(G,), we get 
again the contradiction to (1). 


§ 2. Diagrams containing just two double bounds 


The aim of this chapter is to treat the case that we have a chamber system 
@ whose diagram A contains O——O——>» and all other bonds are single 
bonds. For this chapter we assume that @ fulfills the hypotheses of the theorem. 
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We first treat the case that @ is of rank four. Then the possible diagrams 


are: 
: [| 


(Here we used a result of [18] saying that there are no triangle diagrams but 


) 


(2.1) Lemma. A is not of type 


1 2 3 4 


Proof. Suppose false. We will get a contradiction in a series of steps. By (1.8) 
we know the possible local structures for G4/K,4 and so for G, 4/K,,4. 


(2.1.1) G,/K4 does not satisfy (1.8)(v). 


Suppose false. Then G, 4/K,,4 involves an automorphism group of U;(2) 
and so G,/K, involves an automorphism group of U,;(2) which implies that 
G,.2,4/K,,2,4 involves L,(4), contradicting (1.8)(v). 


(2.1.2) G4/K4 does not satisfy (1.8)(i1i). 


Suppose false. Then we get G,; 4/K,.4=92¢ (2) or O¢ (2) and so G,/K, =Qg (2) 
or Og (2). As K,K4/K, and K,K,/K, are both 2-groups (1.5) implies that 
K, and K, are both 2-groups. Furthermore by (1.8) K, © K4. 

Suppose (,(K,)¢K,. Then Z(K,)¢&Z(K,). Furthermore Q,(Z(S)) 
+ 2,(Z(K,)), since otherwise 2, (Z(K,))=2,(Z(S))=Q,(Z(K4))a<G,, G4> =G, 
a contradiction. 

If J(S)OK,, then Q2,(Z(K,)) is an F-module for G,/K, and thus has to 
be the natural module by (1.1). Now by (1.4) J(K,4)=J(K,)a1<G,, G4> =G, again 
a contradiction. So we get J(S)\SK,SK, and so J(K,)=J(S) 
=J(K,)=1¢<G,, G,>=G, the same contradiction as before. 

Thus (;(K,)9K, and so G,;=K,Cg (K,). We have G,/K,=L;(2)xS3. 
Choose teG,\G,. Then t normalizes 07(G,0G,)K; and so t normalizes 
07(G,9G,). Now (K,K,)'K,/K, is centralized by 07(G,G,) 
=07(G,G,). But G,1G,/K, is the stabilizer of an isotropic 3-space in the 
natural representation of Q;(2) and so O7(G,G,) centralizes no 2-element 
in 0,(G,G,/K,). Thus (K,K,)'SK, and so (Ki, K,)'=K,K,. Then 
K,K,<Xt, G;>=<G2, G,>=G, yielding K, 1K,=1. As |K, K,/K,|=2 we 
get |K,|=2. Let peX,, o(p)=3 such that <p) is normalized by K,. This is 
possible as K, 7 K,. Now [K,, K,]&K,.K,=1 and so [p, K,]=1. But in 
G, we see that <p> K,/K,<1G,/K, and so <p) is normalized by S. From the 
structure of G, we see that X,/K,=2'**(S,xZ,) or 2'*4*(S, xS,). We know 
that K, is a 2-group and so any 3-element of X, is contained in X,\K,. But 
in X,/K,4 there exists no normal subgroup of order 3. 


(2.1.3) G4/K4 does not satisfy (1.8)(iv). 
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Suppose false. We first show that G, 4 involves 9¢ (2). Otherwise we get 
that G,/K,~Sp,(2). By (1.5) we have that K, and K, are {2, 3}-groups. As 
03(K,)SK, and 03(K,4)¢K,, we get 03(K,)=03(K,)=1. Set 0,=0,(K,), i 
=1,4. We have 0,9K,. Set G,=<Q$'>. Then [K,,G,]<Q, as [K;, Q,4] 
£Q0,0K,SQ;. Hence for PeSyl;(K,) we get G;,=K,C¢,(P). But G,; NG, 
involves 3-A, and so Cg ,(P) has to involve a perfect central extension of Sp, (2) 
by a cyclic group of order 3. Such a group does not exist. 

So we have that G,.4/K;.42; (2) or O” (6, 2) and so G,/K, = U,(2). Now 
G,/K, is involved in 2, x Aut(L3(4)) and so both K, K,/K, and K,K,/K, 
are 2-groups. Thus (1.5) yields that K, and K, are 2-groups. Furthermore we 
see in G, that |K,: O,(K,)|<3 (there might be some diagonal automorphism 
of order 3 involved in G,). 

Suppose that (,(K,)¢K; for i=1, 4. If Z,+2,(Z(S)), then [X4, 2,(Z(S))] 
+ 1. Now by [16; (2.2)] we have that Z, is an F,-module for G,/K, with offending 
subgroup in S/K4,, contradicting (1.11). Hence Z,=2,(Z(S)) and so Z, 
+Q,(Z(S)). Now we apply (1.7) to (G,, X,) where X¥,¢X, with S<X, and 
X ,/0,(X,)=S3. It is easy to check that (G,, X,) satisfies the assumption of 
(1.7). As X ,/K4&G4/K4, we have |[p, 0,(X ,/K4)]|= 16 for pe X ,, o(p)=3. Now 
by (1.7) Z, is an F,-module for G,/K, and so by (1.1) Z;/Z,Z(G;) is the 
natural module. But then SU,(2) is involved in G,, contradicting 
G,/K, Saut(U,(2)) and 3 || K,|. 

So we have that €,(K,)GK, or C,(K,) Gd K4. We have that 0? (G, 0 G4/K,) 
is an extension of an extraspecial group of order 2° by an automorphism group 
of U,(2) and 0? (G, 1G,/K,) is an extension of an extraspecial group of order 
27 by the same automorphism group. As |S: K,(K4S)| has to be the same 
viewed in G, and G, we see that K,K,=0,(G,G,), a contradiction to 
[0?(G, AG,), K;]=1 for i=1 or 4. 

Now we can finish the proof of (2.1). We know that G,/K, satisfies (1.8)(i) 
or (ii) and so G,/K, =Sp,(2). This gives again that K, and K, are 2-groups. 


(2.1.4) G4/K4 does not satisfy (1.8)(i). 


Suppose false. Let (,(Kj)@K; for i=1,2. Application of (1.12) gives Z,4 
= 2, (Z(S)). Now we have a contradiction with (1.14). 

So we may assume that either (,(K,)9K, or €;(K4)9K,4. In the latter 
we get |K,K,:K,|S2, as 07(G,G,) centralizes K,K,/K, and G,G,/K, 
is the stabilizer of a point in the natural representation of Sp,(2). Now 
|S: K,K,|22°. As |S:K,|S2° we get |K,:K,QK,|S2 and _ so 
[07(G, 0G,), K,] © K,Kg,. This shows €,(K,)GK, by the 3-subgroup lem- 
ma. 

Hence in any case (,(K,)GK,. As Z(G, \1G4/K,4)=1, we have K,oKg. 
But then [07(G,0G,),K4]&K,, as |K4:K,|S2. This shows G, 
=$<07(G, G4), O?(G30 G4)> =SCg,(K4) and so K,<1<G,, Gg>=G. Hence 
K,=1. Now by (1.8) |K4|=2. 

Now SeSyl,(Sp,(2)) and so S possesses exactly one elementary abelian sub- 
group of order 64 which is 0,(<«X3, X4>). But then 0,(<«X3, X4>)= Kz, contra- 
dicting G,/K,=2, x L;(2). 

To finish the proof we just have to handle the case that G,4/K, satisfies 
(1.8)(ii). In this case 0,(¢X ,, X2>)=O02(<«X 2, X3>)=K,4. Furthermore K, Ky. 
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Now G3/K;2A,xS3, SsxS3; or (A,xZ;)-2. In particular K,¢K, and 
|K4:K3|=2. Choose peX,4, of order 3. Then p normalizes K;K,/K,. As 
K,K,/K, is abelian and <p) is normalized by some element in K,\K3 we 
get [p, K,]<K,. But it is easy to see that in Sp,(2) an element of order 3 
never centralizes an elementary abelian group of order 16. 


(2.2) Lemma. Let A be of type 


then G, ~ G,=Sp,(2), U;(2) or 0, (2). 
Proof. Again we split the proof in different subcases. 
(2.2.1) If G,/K, contains U,(q) or U;(q), then G, =~ G,~=U,(2) or 0, (2): 


As G, is one of the groups in (1.8) we have q=2. Now <X;, X,4)> involves 
L3(4) and so G,/K, involves U,(2) or U,(2), too. By (1.8) G24/K4=G,,4/K4 
and so G,/K,=G,/K, and K,, K,5K,. 

Now the arguments differ a little bit. Let G,/K,~0,(2)%G./K>. We have 
that K, K,/K, is normal in G; 0G,/K,. As G; NG; acts irreducibly on K, >/K,, 
we get |K, K,/K,|=2° or K,=K>, in which case K,=K,=1, which is the 
assertion. In the first case K, K,/K,=O,(K,4/K,) is elementary abelian by the 
structure of G, ,/K, and extraspecial by the structure of G, 4/K,, a contradic- 
tion. 


So assume G,/K, = U,(2)=G,/K,. Again we may assume that K, + K,. Now 
the parabolic in U,(2) corresponding to G,; 1G, possesses the normal structure 
2°/2°/SL3(4). This shows that K, K.,/K, contains an elementary abelian sub- 
group of order 2°. But K,/K, is extraspecial of width 5 and so cannot contain 
such a big elementary abelian subgroup. 


(2.2.2) If G,/K, #0,(2) or U,(2), then G,/K,~G,/K,=Sp,(2). 


We have that G,/K, involves Q; (q) or Sp¢(q). Suppose that G,/K, involves 
Qs; (q). Then by (1.8) q=2 and X, involves L,(4). Now we get that G,/K, 
is as in (1.8)(iv) and so G, 1 G4/K, involves 3-A,. So we get G,/K,=Sp,(2). 

Now both K, K,/K, and K,K,/K, are 2-groups. Thus by (1.5) K, and 
K, are 2-groups. Hence K, K,/K, is a 2-group and by the structure of Sp,(2) 
also K, K,/K, is a 2-group. Now again by (1.5) K, is a 2-group. But G, VG, 
involves 3-A,, a contradiction. 

So both G,/K, and G,/K, involve Sp,(q). Then by (1.8) q=2 and so 
G,/K, =G2/K,=Sp¢(2). 


(2.2.3) If G,/K, ~G2/K,Sp¢(2) then G, = G2 =Sp.(2). 


By (1.5) K, and K, are 2-groups. Now B is a 2-group and so all K; are 
2-groups. Furthermore G,/K, is as in (1.8)(i) or (ii). 

Suppose that G,/K, satisfies (1.8)(i). Let C,(K,) 9 K,. As Z(G, NG,/K,)=1, 
we get K, ©K, and so K, = K,=1, the assertion. 

So we may assume that (,(K,)¢K; for i=1, 2. As |Z(G, \G,/K,)|=2, we 
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get C;(K,)&K, or |K,K,:K,|S2. In the latter K, °K, and so €,(K,)9K,, 
a contradiction. 

So €;(K;,)SK; for i=1, 2, 4. Application of (1.12) gives Z,=Q,(Z(S)). Now 
we get a contradiction with (1.14) applied to the pair (G,, G,). 

We are left with the case that G,/K, satisfies (1.8)(ii). Then Kz =K, 4=K) 4. 
Further K, ° K,>K>,. We have that K, K,/K,<1G,G,/K, and so K, K,/K, 
is abelian. Suppose K,9/K,. Then K,,5K,K,SKz,4, a contradiction to 
K,,.2/K,; 9K, ,4/K,. Thus K,=K,=1 and we have G; >G,~Sp,(2), the asser- 
tion. 

Now (2.2) follows from (2.2.1){2.2.3). 


(2.3) Lemma. A is not of type 


Proof. 
(2.3.1) G,/K, does not satisfy (1.8)(v). 


Suppose false. Then G,/K, involves U,(2). Hence X ,/K, involves L,(4) while 
by (1.8) X3/K, involves L,(4), a contradiction. 


(2.3.2) G,/K, does not satisfy (1.8)(iv). 


Suppose false. Then we may assume that G, »/K, , involves 2¢ (2) and so 


G,/K,=U,(2) since by (1.8) X, involves L,(4). In particular G,,,/K>,, is an 
automorphism group of L,(4). This shows that G23 4/K 3.4 involves L,(4). 
But on the other hand G, 4/K,,4 involves A, and so G4/K,=Sp,(2). This shows 
G2,3,4/K2,3,4=5S3, a contradiction. 


(2.3.3) G,/K, does not satisfy (1.8)(iii). 

Suppose false. Then G,/K, and G,/K, both involve Q,; (2). Furthermore 
K,, K,SK, and K,/K,, K,/K4, are both 2-groups. By (1.5) we get that K,, 
K, and K, are 2-groups. 

Suppose C;(K,)SK3. If 2,(Z(S))=2,(Z(K2)), then 2,(Z(K,))=2,(Z(S)) 
and so 2, (Z(S))=1<G,, G,>, a contradiction. Thus 2, (Z(S))+2,(Z(K,)). Now 
J(K,)SK, by (1.4) and so J(K,)=J(K,)1¢G,, G,> =G, again a contradiction. 

So we have (;(Kj)9K; for i=2,4. But then K,0K4<1¢K;, Cg,(K))|i 
=2,4>=G and thus K,nK,=1. Hence |K,K,/K;|S2, i=2,4, since G; 
=K;C,,(K;) we obtain |K;|<2, i=2,4. This gives 2°<|K,|<2’. We have 
G3/K;=A,. This shows K,¢K, and so G3;0G,/K,;=Eg,L;3(2). Then 
G,G,/K, possesses a factorgroup E, L;(2), which contradicts [15; (1.12)]. 


(2.3.4) G,/K, does not satisfy (1.8)(i). 


Suppose false. Then we get that G,/K,~G,4/K4=Sp,(2). Thus by (1.5) K>, 
K,, are 2-groups and so by (1.6) all K; are 2-groups. Claim €,(K;) K;, i= 1, 2, 4. 
Suppose this is false. If C;(K,)9K,, then as Z(G, 0G,/K,)=1, we get that 
K,¢K,. Now |K,:K,|S2 and so €,(K,)9GK,. The same applies for 
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C;(K,)9K,. So without loss (,(K,)GK,. Then as |Z(G,G,/K,)|=2, we 
get |K, K,: K,|S2 and so again K,&K, and €,(K,)9K>. 

So we may assume K,© K,>K, and (,(K,)9K,. Hence K,=K,=1. Now 
|K,|=8 and as G;/K,;=Ag, we get (,(K3)9K;3, contradicting the structure 
of Sp¢(2) (Cg, .¢,(K3)=02(G2 0 G3)!). 

We have €,(K,)¢K; for i=1, 2,4. By (1.12) we have that Z, =Q,(Z(S)). 
Now the pair (G,, G,) fulfills the assumptions of (1.14). This yields a contradic- 
tion. 

To finish the proof of (2.3) we just have to handle the case that G,/K, 
is as in (1.8)(ii). Assume this is the case. Then G,/K,~G,/K,=Sp,(2) and all 
K; are 2-groups. Furthermore G3/K3;~A,. Now the structure of G3; shows 
G,,3/K,,3=S3 xS3, but the structure of G, shows G, 3/K,,,=(Z; x Z;)-2 by 
(1.8), a contradiction. 


(2.4) Proposition. If A is connected and contains O—=O— and all other 
bonds are single bonds, then either |A|=3 and G satisfies (1.8) or A is of the 
shape 


2 


and G, ~ G2 =Sp2(n-1)(2), O4- 1)(2) or O,,-1(2). 


Proof. First we show that A is as in the assertion. We prove (2.4) by induction 
|A4|. We may assume that |4|>3. Now let i be some vertex of A such that 


the residual diagram A; is connected and contains c——o——».. If |4;|=3, 
then |4|=4 and the assertion follows with (2.1)(2.3). So assume |4;|24. By 
induction we may assume that 4; is of type 


1 


2 


Suppose i is connected with j+n—1, then A,-, is connected and contains 
a—no—D , so it has to be of the same shape as 4;, which is only possible 
if j=n—2. Now A is of shape 


Application of [13] to A, gives a contradiction. 

So we have proved that the diagrams are as in the assertion. We now deter- 
mine the groups for | A|>4. 

By (2.2) we have Gy\5._n/Ky,5,...2%SPo(2), O,(2) or O,(2). This gives 
G,/K, =Spo(n- 2)(2), Urn —2)(2) or Uz,~ 1 (2). The same holds for G,/K ,. By induc- 
tion we have K,=K,,=K,,, and so K,<K,>K,. We have that K, K,/K, 
and K, K,/K, are {2, 3}-groups and so by (1.5) K, and K, are {2, 3}-groups. 
As 03(G,0G,/K,)=1=03(G,G,/K,), we have that O;(K,)©K, and 
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03(K,)SK, and so 03(K,)=03(K,)=1. As 0,(K, K,/K,)aG,G,/K, and 
0,(K, K,/K,)¢K,/K, we get O,(K, K,/K,)=1 (Sp,(2), U,(2) do not possess 
such nontrivial subgroups which can be checked using the Steinberg relations). 
But then O0,(K,)&K, and O,(K,)¢K,. This gives O,(K,)=0,(K,)=1 and 
so as O3(K;,)=1 and K; is a {2, 3}-group, we get K;=1 for i=1, 2. 


§ 3. Proof of the Theorem 


We again assume that @ fulfills the hypotheses of the theorem. 


(3.1) Lemma. A is not of type 





1 2 3 4 


Proof. We have that G,/K, and G,/K, satisfy (1.8). If G,/K, satisfies (1.8)(v), 
then G, ,/K,, involves U;(2) and so G4/K,4 satisfies (1.8)(v). But from the struc- 
ture of G, we get that X,/K, 4 involves L,(4) while from the structure of 
G, we get that it does not. Next we show 


(3.1.1) G,/K, does not satisfy (1.8)(iv). 


We will prove (3.1.1) in a series of steps depending on the structure of G,. 
For this we assume that G,/K, satisfies (1.8)(iv). 


(1) G,/K, does not satisfy (1.8)(iv). 


Suppose false. Then we get that G, ,/K,,. and G3 4/K3,, both involve U,(2). 
This shows that U,(2)x L,(4)SG,/K,<Aut(U,(2)xL,(4)) and U,(2) 
x L3(4)<G3/K3<Aut(U,(2) x L,(4)). Hence K,K3/K, and K,K;/K; are both 
elementary abelian of order 16. By (1.5) K, and K; are 2-groups. Now we 
have the following: If G,,./K,..=U,(2) then by (1.8) G,3/K,3;=A. and so 
again by (1.8) G3 4/K3 4=U,(2). Thus in this case G,/K,=G3/K,3 = U,(2) x As. 

If G,.2/K,.22=U,(2):2 then by (18) G,3/K,3;=S, and _ so 
G3.4/K3,4= U,(2)-2, hence G,/K,~G3/K,=(U,(2) x L(4))-2. So in any case K, 
and K, are 2-closed. Set Q;= 0,(K;), i=1, 4. Then | K,/Q;|=3. 

Claim €,(Q;)<Q;, i=1, 4. Suppose this is false. If C,(Q,)9Q,, then Q, <K, 
as Z(G, 0G,/K,4)=1. So Q,5Q4. As |Q;|=|Q4| we have Q,=Q,=1. Now 
K,;<1G,0G;. As 0,(G,G;) is extraspecial of order 2’, we get |K3;|=2 or 
27. As |S|=2!? or 2!*, and G;/K,=U,(2) x L,(4) or (U,(2) x L,(4))-2 we have 
a contradiction. 

So we have €,(Q,;)<Q,, i= 1, 4. As by (1.11) G/K; do not possess F-modules 
with offending subgroup in S/K;, we get 2,(Z(S))a1G, and 2,(Z(S))</G,. Now 
we apply [16; (2.2)] to the pair (X,4, G,). This yields that Z, is an F,-module 
for G, with offending subgroup in S/K;, a contradiction to (1.11). 


(2) G4/K4 does not satisfy (1.8)(iii). 
Suppose G,/K, satisfies (1.8)(iii). Now K,&K, and K,/K, is a 2-group. 
Then by (1.5) K, is a 2-group. If €,(K,)GK,, then Z(K,)&Z(K,) and so 


2Q,(Z(K,))+2,(Z(S)). Now by (1.11) J(K4)=J(K,)a1<G,, G4> =G, a contradic- 
tion. 
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So we may assume (€(K,)9/K,. We have Z; x AgSG,/K,52, x 2X5. Let 
teG,\G,. Then t normalizes O?(G,G,) and so (K,K,)' is centralized by 
07(G,G,). But Z(G, G,/K,)=1 and so (K,K,)'SK,K;. This shows 
K,0K,<1<G,, 07(G, A G4)> =<G2, G,>=G. Hence K; 1 K,=1, and so |K,| 
<2. We have [K,, K,]=1. 

Claim K,=1. Suppose this is false. Then K, inverts some pe X,, 0(p)=3. 
Hence [p, K,]=1. As <p, K,)<1G,, we get that <p)<1.X,, but X,/K,4 is a 
parabolic in U,(2) and U,(2) possesses no element of order 3 normalized by 
a Sylow 2-subgroup. 

So we have K,=1. Hence K, is extraspecial of order 27. Now G30G,/K4 
acts nontrivially on K, and so it acts irreducibly on K,4/Z(K,). This shows 
K,=K3<1<G;3, G,>=G, a contradiction. 

Now we can finish the proof of (3.1.1). We have that G,/K, is involved 
in S$; x U,(2)-2. Thus both K, K,/K, and K,K,/K, are 2-groups. In particular 
K, and K, are 2-groups by (1.5). 

Let €,(K,)&K,. Then by (1.12) Z,=,(Z(S)). Now application of [15; 
(2.9)] yields a module V for G,/K, such that |[V, t]|<2° for teZ(S/K,). This 
contradicts (1.11). 

We have (;(K,)9/K,. If G4/K, satisfies (1.8)(i) then as Z(G, 0G4/K,4)=1 
we get K,©K,. But G,;G,/K, possesses no factor group isomorphic to 
G,0G,/K, by (1.8). So we have that G,/K, is as in (1.8)(1i). Now G,/K,=S, 
x U,(2), S3 x U,(2)-2 or (Z3 x U,(2))-2. In any case G, 4/K2,4=53 x S3. But from 
the structure of G, we get with (1.8) that G, 4/K,,4=(Z; x Z3)-2, a contradiction. 


(3.1.2) G,/K, does not satisfy (1.8) (iii). 


Suppose false. Then we apply (3.1.1) to G,/K,. This shows that G,/K, satisfies 
(1.8)(iii). This implies that X,/K,K, involves L,(4) while the structure of G, 
implies that X,/K,K, does not involve L, (4). 

By (3.1.1) and (3.1.2) we have that G,/K, and G,/K, satisfy (1.8)(i) or (ii). 
In particular by (1.6) all K; are 2 groups. 


(3.1.3) G,/K, does not satisfy (1.8)(ii). 


We have that G;/K,;~S,;xA,, S3xS, or (Z,;xA¢)-2. In any case 
G,,3/K,,3=53;xS3. But by (1.8) the structure of G, implies that 
G, 3/K,,3=(Z; x Z3)-2, a contradiction. 

So we have shown that both G,/K, and G,/K, satisfy (1.8)(i). Claim 
C;(K;)JK; for i=1 or 4. Suppose this is false. Then we may assume Z, 
+, (Z(S)). Application of (1.12) gives a contradiction. 

Without loss €,(K,)9K,. As Z(G,;AG,4/K,)=1 by (1.8), we get that 
K,&Ky,. Now K,4/K,<1G,G,/K, and so by (1.8) either K,=K, or |S: K,| 
<2°. But we know |S: K,|=2’. This shows K,=K,=1. Now by (1.8)(i) we 
have G,,35@C(Z(S)) and G,,,¢C(Z(S)). But then G=<G,,3, G2,4>< C€(Z(S)), 
a contradiction. 


(3.2) Lemma. A is not of shape 
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Proof. Suppose false. If G,/K, satisfies (1.8)(v), then also G,/K, satisfies (1.8)(v). 
But comparing the structure of X,/K, and X,/K, gives a contradiction. 


(3.2.1) G,/K, does not satisfy (1.8)(iv). 
Suppose this is false. We first show 
(1) G,/K, does not satisfy (1.8)(iii). 


Suppose this is false. Then G,./K,,.=U,(2) or U,(2)-2. Furthermore by 
(1.8) Gz 1.4/K2,1,4=53 and G23.4/K2,3,4=53. This shows G,/K,=Sp,(2). Now 
K,SK, and K,/K, is a 2-group. Thus we have that K, is a 2-group by (1.5). 
Furthermore | K,/K,|=2'°-3 and so K, is 2-closed with | K,/0,(K,)|=3. Then 
G,/0,(K4)=Z; x Sp,(2), as there is no 3-part in the Schur multiplier of Sp, (2). 
But this contradicts the fact that G, 0 G, involves 3- A,. This proves (1). 

(2) G,/K, does not satisfy (1.8)(iv). 

Suppose false. Then G,,34/Ki,3,.4=L2(4) or L,(4)-2. This gives 
G,/K,=0,(2) and G3/K,~0,(2). Now K3,K,/K, and K;K,/K; are 2-groups. 
Thus by (1.5) K; and K, are 2-groups. Now we get that K, K;3/K, is contained 
in the point stabilizer of U;(2) and so K,K;/K; is t-cdoeed with 3-part of 
order at most 3. This shows that K, is 2-closed and |K,:0,(K,)|<3. The 
same holds for K,. Set 0;=0,(K;), i=1, 2. 

Claim €;(Q;)9Q; for i=1, 2 or 2. Suppose this is false. As by (1.11) neither 
G,/K, nor G,/K, possess an F-module, we may assume that Z,=2,(Z(S)). 
Then Z;+2,(Z(S)) and so we have that Z; is an F-module. By (1.1) Z; is 
the natural module. Then by (1.3) 2, (Z(S))a1¢X,, X4, G,> =G, a contradiction. 

Without loss €;(Q,)9Q,. As Z(G, OG,/K,)=1, we get 0,<K, and so 


0,=Q,=1. Hence |S|<2'*. But the structure of 0,(2) shows |S: K3|2=2'°, a 
contradiction. This proves (2). 

By (1) and (2) we have G,/K, satisfies (1.8)(i) or (ii). In these cases 
G2,3,4/K2,3,4=53. On the other hand G, 3 4/K;,3,4=L2(4) or L,(4)-2. So 
G,/K4= Us(2). Then the structure of G, implies G2 3 4/K>3.4=G,,3,4/K1,3,4; 
a contradiction. 


(3.2.2) G,/K, and G,/K, satisfy (1.8)(i) or (1.8)(ii). 


Suppose false, then by (3.2.1) we may assume that G,/K, satisfies (1.8)(iii). 
Now G,,>.4/K,,2,4=L2(4) or L,(4)-2 and so by (1.8) G,/K, satisfies (1.8)(v), 
which contradicts (3.2.1). 

Now we finish the proof. By (1.6) we have that all K; are 2-groups. Claim 
C,;(K;) 9K; for i=1 or 2. Suppose this is false. If G,/K, and G,/K, both satisfy 
(1.8)(i), then by (1.12) we get that 2, (Z(S))a<G,, G,> =G, a contradiction. 

So without loss G,/K, satisfies (1.8)(ii). Then K,< K, and so G,/K, satisfies 
(1.8)(i). Otherwise K, =K,=1 and so |S|<2*, but as G;/K,=Sp,(2) we know 
|S|=2°. Let Q,(Z(K>2))+Q,(Z(S)), then by (1.10) J(K,)=J(K,)a<G,, G.>=G, 
a contradiction. Thus 2,(Z(K,))=2,(Z(S)). Now 2,(Z(K,))SQ,(Z(K3)) 
=2,(Z(S)) and so Q,(Z(K,))=2,(Z(K))a1<G,, G,> =G, a contradiction. 

So without loss €;(K,)GK,. As G4/K4=Sp,(2) we see |Z(G, 0 G4/K4)|=2 
and so |K, K,/K,4|S2. In particular |S: K,|=2°® and so G,/K, satisfies (1.8)(i). 
But as |S:K,K,|=2°, and K, K,<1G,G,, we see with (1.8), that K,oK, 
and so (,(K,4)GK,. Now K,<1<0?(X,), G,>=G. This shows K,=1. Then 
G;=G,=Sp,(2) and so |K,|=2. By (1.8) applied to G,, we get that 
G,,3SN(Q,(Z(S))). But N(Q,(Z(S))) 0 G3=G,, 2,3, a contradiction. 
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(3.3) Lemma. If A is a square diagram, then A is of shape 


2 3 














i 4 


Furthermore one of the following is true 

(i) G;,/K;,;=A6 or So for {i,j} €{(1, 2), (2, 3), (3, 4), (1, 4)} K,=1 for some 
ke{1, 2, 3, 4}. 

(ii) G,, ;/K;,;=U,(2) or U,(2)-2 for (i, j)=(1, 2) and (1, 4). K,=1=K, and G,, 
G, satisfy (1.8)(iv). 


Proof. That A has the required shape follows from (3.2). Suppose first that 
all G;/K; satisfy (1.8)(i) or (ii). All we have to show is that there is some i 
such that K;=1. By (1.6) all K; are 2-groups. If G,/K, and G,/K, both satisfy 
(1.8)(ii), then we get K, =K,=1. So to prove (i) we just have to handle the 
case that G,/K, satisfies (1.8)(i). 

By the same argument we may assume that G,/K, satisfies (1.8)(i). Let 
(,(K,)&K,. By (1.12) we have 2,(Z(S))<iG, and so €,(K3)7K;. Now 
2,(Z(S)) 1K3<1¢G,, Gz>=G, which implies K,;=1. But then (€,(K,)9K,, a 
contradiction. 

So it remains to treat (,(K,)9K; for i=1, 3. As Ki; 1K3<1¢G,, G3), we 
get that K, 0K,=1. Furthermore K, K3/K,<1G, 0G; and so |K, K3: K,|S2. 
This shows |K,|S2=2|K,|. So we may assume |K,|=2=|K,| otherwise we 
are done. As Z(G, 1G,/K,)=1, we get K, ©K, and so G, and G, satisfy (1.8)(ii). 
In particular G, 4/K,,4=(Z; x Z3):2. Now we have G, ,5C,,(Q,(Z(S))) and 
Q,(Z(S))=K,K3. This implies that Cx,(X,)=K3; and Cx,(X3)=K,. Hence 
<(K, K3)°*>=E is of order 16 and [E, K,,,]=1. But (<K,,K,>¢K,, and 
C;(K,)=K,, €s;(K4)=K,4. In particular E©K,K,4. But the size of this inter- 
section can be determined in G, and so we get |K,7K,|=8, a contradiction. 
So we have proved (i). 

Next we treat the case that one of the groups G,;/K; does not satisfy (1.8)(i) 
or (ii). So without loss G,./K,,. involves U;(2) or U,(2). In the former we 
get that both G,/K, and G,/K, satisfy (1.8)(v). This is impossible. 

We have that G, ,/K,,, involves U,(2). Again it is impossible that both 
G,/K, and G,/K, satisfy (1.8)(iii). So we may assume that G,/K, satisfies (1.8) (iv). 
This implies that G,/K, satisfies (1.8)(i1i) and so G,/K, satisfies (1.8)(iv). 

To prove (ii) we have to prove that K,=K,=1. We know K,¢K,2>K,. 
Furthermore K,/K, and K,/K, are 2-groups. Then by (1.5) K,, K,, K4 are 
2-groups. 

Application of (1.14) shows that €,(K,)9GK, and €,(K4)9K, and so 
K,0K,=1. By (1.8) G, G,/K, is an extension of a special group E of order 
2'° by a group H with A,xS$,;¢CH¢S,~xS3, and |Z(E)|=4. As G,=SC,(K,), 
we get [07(G,G,), K4]=1. But Z(G,0G,/K,)=1. So K,=K,4=1. This 
proves (ii). 


(3.4) Proposition. If 4 contains O——oO——® then the theorem holds. 


Proof. By (2.4) we may assume that the diagram A contains at least three double 
bonds. First we show 


(3.4.1) A is one of the diagrams of the theorem. 
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Choose ie A such that A; is connected and contains 0——o-——.. By induc- 
tion A; belongs to the list of the theorem. Now we check the possibilities: 


n+m+1 


Let i be connected with xe{m+2, m, ...,n—1}. Then A, is of the same type 
as A; and so x=n—1 and A looks like 


Now 4, is of type 
a contradiction. 


Suppose now that i is not connected with n. Then 4, looks like 
1 


for n+m+2 


and so A is in the list. 
Assume n=m-+2. If i is connected with m+1, then by [18] 4 is in the 
list or A looks like 


i 


But now 4, is not in the list, a contradiction. 
Assume now that i is connected with n. Now we look at 4,. As 4, has 
to be in the list we get A, is of type 


2 


We have that i is not connected with 2. Now looking at 4, shows that 
1 


i is not connected with 1 and so A is of type : O——o in parti- 
cular A is in the list. n i 


m 
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Thus i is connected with 1. As n+ 2, (3.3) implies n= 4. Now A, _ , is connected 
and so A,-_, is a line. But there are no lines containing two double bonds 
and more than 3 nodes in the list. This handles (1). 


eles See 


(2) A; is a star 2 : 
3 4 


Let i be connected with xe{2,...,n}, wlog. x=2. If n=2, then 4 
is in the list. Thus n>2. By [18] i is not connected with 1. Suppose that i 
is not connected to any ye{3,...,n}. If n>3, then A, looks like 
—o—o—_ or 0 Oo 1e © both are not contained in the list. 
Thus n=3 and A looks like &—o-—"D——_-0 or D&D oO 0, which 
contradicts (2.1) and (3.1). 

Thus we may assume that i is connected with 3. Now A, is in the list. 
Let n be at least 4. Then A, is a complete bipartite graph and so A is a complete 
bipartite graph with partitions {1, i} U {2, ..., n}. 

Let n=3. Then A is a square and so by (3.3) A is in the list. 

It remains the case that i is connected with 1 and no other node of 4;. 
Let n> 3, then 4, contains at least two double bonds and so A is a star containing 
only double bonds. Let n=3, then A is 


2 
y. both are in the list. 
3 


(3) A; is a complete bipartite graph containing only double bonds, A; is not 
a star. 

Let [, UT, be the partition of A;. Let xe, x connected with i. Let ye, \{x}. 
Then A, is a complete bipartite graph containing only double bounds or 4, 
looks like 














Suppose the latter. Then |/,;|=2=|J,|. Now A, is a square diagram or of 
type O_O —_© which contradicts (2.1) and (3.3). 


Let A, be a complete bipartite graph containing only double bonds. Then 
i is connected with no element in I, and every element in [,\{y}. If |I,|>2, 
then we look at A, and get that A is a complete bipartite graph and so A 
is in the list. 

Let |I,|=2. If i is connected with y we may argue as before. But now we 
have that A, is a star and we get the assertion with (2). 

It remains to prove the assertion about the groups. 


(3.4.2) The theorem is true if A is of type 
1 


n+m+1, m=2: 
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By (2.4) we may assume m2 3. Now by induction K,=K,_,,=K,=1. Thus 


G,E{Sporn—m+1}(2)s Ozin—my+13(2)> Ooyn—my+11+1(2)} for any IC{1, ..., m}, |I| 
=m—l. 


(3.4.3) The theorem is true if A is a star 


By (1.8) we may assume that n=4. We will argue by induction on n. 

Let n=4. Assume that no G,/K,, J&{2, 3, 4}, |J|=2, involves Ag or S¢. 
Then by (1.8) K3 4=K2.4=K2,3<1¢X,, X2, X3, X4>=G, the assertion. 

Assume now that any G,/K,, J&{2, 3, 4}, |J|=2 involves A, or S,. Then 
all G,/K;, ie {2, 3, 4} satisfy (1.8)(i) or (ii). Further by (1.6) all K; are 2-groups. 
If G,/K, and G;/K, both satisfy (1.8)(ii), then K,=K,=1. So we may assume 
that G,/K, satisfies (1.8)(i). Now by (1.14) we get that €,(K,))9K; for i=2, 
3. As Z(G,0G;/K,)=1, this implies K, = K,=1. 

So it remains the case that G,/K, satisfies (1.8)(iv). Now by (1.8) we see 
that G3/K; satisfies (1.8)(iv). Now K,K;3/K,; and K,K;/K, are 2-groups and 
so K,, K3 are 2-groups by (1.5). Again (1.14) implies that C,(K;) 9K; for i=2, 
3. As |Z(G,0G3/K,)|=2 we get |K3K,/K,|S2. Furthermore 
K,0K3;<1<G,, G;>=G, and so K, 1 K;=1. This shows |K,|<522|K3]. 

We have to show | K,|=1=|K3]. Suppose this is false. As G, 4/K > 4 possesses 
no factorgroup 2*-A, or 2*-S,, we get that G,/K, satisfies (1.8)(ii). So K, 
=K,4=K34. Then Z(K,) is of order 32. Now C2,(G2,4)=Kz2 while 
C2(x,)(G3,4)=K3. This shows K,~ K; in G, and so K, possesses 31 conjugates 
under Gy, Then G,/C€,,(Z(K4))=L;(2). Set Q,=O0O,(K,4). Then 
G,/Cg,(Q4/Z(K,4)) is a subgroup of L¢(2) involving L;(2) and 3-A,. This implies 
G,4/Cg,(Q4/Z(K4)) = L¢(2). Set Gg=G4/Cg,(Q4). Then G, is an extension of a 
2-group by L,(2), contradicting the fact that G4/Cg,(Z(K4))= Ls (2). 

So we have shown that the assertion holds for n=4. By induction we have 
K,=K, for some J GI, |J|=|I|—3, 2eJ. Without loss neJ. Then K, °K,=K3. 
If we choose K, of minimal order in {K;|ie{2, ..., n}}, then we get K,=K,=1, 
which is the assertion. 


(3.4.4) The theorem holds if A is a complete bipartite graph containing only 
double bonds, which is not a star. 


Let I, UT, be the partition of the nodes of 4. By assumption we have {J;| 
>2<|I|. By (3.3) we also may assume |J,|=3. For any ieI we have that 
A; is a complete bipartite graph. If ieIj, j=1 or 2, with |J;|>2, then 4; is 
not a star. There are two cases. 


Case 1. For any ieI,, we have that 4;_,,(c) satisfies Theorem (2)(a). Let kel, 
jelh, then choose ieI, — {k} and so k, je Aj, ie. Gy_¢, a/Ky- 4, = Ae OF Se- 


Case 2. There is some ie, such that 4,-_,,(c) is a chambersystem which satisfies 
Theorem (2)(c). Then A; possesses a distinguished point leJ;, j=1 or 2, such 
that) § Gy-¢1,4/Ki-a.yj=26(2) or O¢(2) for any kel,_;—{i} and 
G,~15,5/Kr-1s,3% Ae OF Sg for any keT\{i, 1}, tel, _j;\{i}. As |[,|23 there is 
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some xeJ,\{1,j} and so A;~_;,,(c) satisfies Theorem 2(c). This shows that 
G,~¢1,6/Ki-a.42=2%6 (2) or O¢(2) for any kel3_;. Furthermore 
Gy ~¢s,5/K1~ (5,3 A6 OF Sg for any sel, tel. 

To prove the theorem it remains to show that there is some J CI, | —J|= 
(i¢J in Case 2) such that K,=1. For this we choose rel (reJ—{1} in Case 2) 
such that K, is minimal. Then 4, _ ,,,(c) satisfies Theorem (2)(a) or (c). In particu- 
lar there is some JCI, reJ (1¢J in Case 2), such that K,=K,. As |I|=5 there 
is some teJ\{r}. Then K,=K,2K,. By minimality we get K;=K,=K,=1, 
the assertion. 


(3.5) Proof of the theorem. By (3.4) we may assume that O——O-——n is not 
contained in 4. Suppose that A contains no triangle. Then any connected rank 3 
diagram is spherical by [18]. Now we may apply [13]. If 4 is not spherical 
then we see that we always have some J&T such that 4,_,(c) belongs to 
O——_O——-O or @————0 and G,,/K , 2 A,, a contradiction. So we have 
that A is spherical and G is an extension of a group of Lie type by diagonal 
or field automorphisms. 


Suppose now that Sf \oit a subdiagram of 4. Choose 4, such that A, 
contains Sg and is connected. By [17] 4, is a complete graph. Now for 


some i€4,, we have that A; is connected and contains a double bond. This 
implies that A is of shape 


But by [18] applied to G,, we get that G, 4/K,,4 is a solvable group, while 
the structure of G4/K4, implies that G, 4/K,.4 involves L3(q), a nonsolvable 
group. This contradiction proves the theorem. 


§ 4. Geometries 


We are going to prove the corollary. First of all we reduce the problem to 
a problem about chamber system. 


(4.1) Lemma. Suppose I is a classical, locally finite Tits geometry over I with 
diagram A=A(I) and flag-transitive automorphism group G. Then we form the 
chambersystem @(I) in the following way: The chambers are the flags of T of 
type I, two such chambers being i-adjacent if and only if they coincide in a flag 
of type I—{i}. Then @(I) is a connected classical, locally finite Tits chamber 
system over I with diagram A and G is chambertransitive on @ (I). 


Proof [20]. 
Let aitabile Masel or aah ceca be a subdiagram of A. If never 


(X;, Xj, x kD just isiéinitn A; on the avant chambersystem the corollary 
follows from (4.1) and the theorem. In the remaining cases we need [3] and 
[14] to prove the corollary. 

But there is a little difference in the conclusion of the corollary and the 
theorem in [14]. Hence we show following an idea of Th. Meixner. 





Chamber Systems 


(4.2) Lemma. There is no classical, finite Tits geometry with diagram A: 


2 


1 


and flag transitive automorphism group G such that G;/K;~ A; for i=1, 2. 


Proof. By (14; (2.3)(a)] we know G,~A,=G, and G; is an extension of a 
group of odd order by Ag. We have G,; =<X>, X3, X4>. But as can be checked 
in Aj, we may generate G, in the following way: 


G,=<2,, X3, X44), 


where (X,, X,)>=L;(2) and <X,, X;)=X, X;3. Similarly we get 


G,=(2%,, X35 Xa) 


with <X,, X,>=L,(2) and (X,, X;) =X, X3. Now we determine the structure 
of <X,, X,). 

We may choose X,, X, inside of G,. But G,=X,X,X, and so 
|G,: X, X;|<3. This shows G,=X, X¥,X, or ¥, X,;=X,X;. In the latter G, 
=G,, a contradiction. Thus we have G,= X , X, X, and so as Sylow 3-subgroups 
of G, are abelian, (X,, X,)=X, X). 

Hence {X,, X,, X3, X,} forms a classical finite Tits chamber system with 
diagram 

2 


3 4 


of type D,. By [17; Theorem(3.1)] the group <X,, X,,X3,X4) 
=<X,, X,, X3, X4>=G is an extension of Q; (2) by diagonal automorphisms. 
But by [14; (2.3)(a)] there is a normal subgroup K of odd order in G such 
that G/K = A,, a contradiction. 


(4.3) Corollary. The cases (e) and (f) in the theorem of [14] do not occur. 


Our corollary follows immediately from the theorem, [3] and [14]. That 
G=U,(5) in the first two cases of the Corollary (4) follows from [12]. 
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